Because of mathjax bug

The Duckietown Book (Duckiebook)

✎🔗

From kits of parts to an autonomous fleet
in 857 easy steps
without hiding anything

The last version of this "duckiebook" and other related documents are available at the URL
http://book.duckietown.org/

For searching, see the one-page version.

Table of contents

✎🔗
Because of mathjax bug
Because of mathjax bug

The Duckietown project

✎🔗

What is Duckietown?

✎🔗

Duckietown history and future

✎🔗

Duckietown classes

✎🔗

Duckietown is an international effort. Many educational institutions have adopted the platform and used to teach robotics and related subjects. If you have used Duckietown in any of your course and cannot find a mention to it here, contact us.

add contact email

Here, we provide a chronologically ordered compendium of the Duckietown related learning experiences worldwide.

2016

✎🔗

Duckietown was created in 2016.

Massachusetts Institute of Technology

✎🔗

Location: United States of America, Cambridge, Massachusetts

Course title: 2.166 Vehicle Autonomy

Instructors: plenty

Educational Level: Graduate

Time period:

Link:

Highlights: Role-playing experience (Duckietown Engineering Co.), public demo

Summary:

Duckietown at MIT in 2016

National Chiao Tung University

✎🔗

Location: Hsinchu, Taiwan

Course title: Robotic Vision

Instructors: Prof. Nick Wang

Educational Level: Graduate

Time period:

Link:

Summary:

Highlights:

Tsinghua University

✎🔗

Location:

Course title:

Instructors:

Educational Level:

Time period:

Link:

Summary:

Highlights:

Rennselaer Polytechnic Institute

✎🔗

Location:

Course title:

Instructors:

Educational Level:

Time period:

Link:

Summary:

Highlights:

First steps

✎🔗

Duckietown for instructors

✎🔗

to write

Duckietown for self-guided learners

✎🔗

to write

Introduction for companies

✎🔗

to write

Frequently Asked Questions

✎🔗

Duckumentation documentation

✎🔗
Because of mathjax bug

Contributing to the documentation

✎🔗

Installing the documentation system

✎🔗

In the following, we are going to assume that the documentation system is installed in ~/duckuments. However, it can be installed anywhere.

We are also going to assume that you have setup a Github account with working public keys.

Basic SSH config.

Key pair creation.

Adding public key on Github.

We are also going to assume that you have installed the duckietown/software in ~/duckietown.

Compiling the documentation (updated Sep 12)

✎🔗

Make sure you have deployed and activated the virtual environment. You can check this by checking which python is active:

$ which python
/home/user/duckuments/deploy/bin/python

To compile the master versions of the docs, run:

$ make master-clean master

To see the result, open the file

./duckuments-dist/master/duckiebook/index.html

If you want to do incremental compilation, you can omit the clean and just use:

$ make master

This will be faster. However, sometimes it might get confused. At that point, do make master-clean.

Compiling the Fall 2017 version only (introduced Sep 12)

✎🔗

To compile the Fall 2017 versions of the docs, run:

$ make fall2017-clean fall2017

To see the result, open the file

./duckuments-dist/master/duckiebook/index.html

For incremental compilation, use:

$ make fall2017

Single-file compilation

✎🔗

There is also the option to compile one single file.

To do this, use:

$ ./compile-single path to .md file

This is the fastest way to see the results of the editing; however, there are limitations:

  • no links to other sections will work.
  • not all images might be found.

The workflow to edit documentation (updated Sep 12)

✎🔗

This is the basic workflow:

  1. Create a branch called yourname-branch in the duckuments repository.
  2. Edit the Markdown in the yourname-branch branch.
  3. Run make master to make sure it compiles.
  4. Commit the Markdown and push on the yourname-branch branch.
  5. Create a pull request.
  6. Tag the group duckietown/gardeners.

Create a pull request from the command-line using hub.

Basic Markduck guide

✎🔗

The Duckiebook is written in Markduck, a Markdown dialect.

It supports many features that make it possible to create publication-worthy materials.

Figures

✎🔗

For any element, adding an attribute called figure-id with value fig:figure ID or tab:table ID will create a figure that wraps the element.

For example:

<div figure-id="fig:figure ID">
    figure content
</div>

It will create HMTL of the form:

<div id='fig:code-wrap' class='generated-figure-wrap'>
    <figure id='fig:figure ID' class='generated-figure'>
        <div>
            figure content
        </div>
    </figure>
</div>

To add a caption, add an attribute figure-caption:

<div figure-id="fig:figure ID" figure-caption="This is my caption">
    figure content
</div>

Alternatively, you can put anywhere an element figcaption with ID figure id:caption:

<element figure-id="fig:figure ID">
    figure content
</element>

<figcaption id='fig:figure ID:caption'>
    This the caption figure.
</figcaption>

To refer to the figure, use an empty link:

Please see [](#fig:figure ID).

The code will put a reference to “Figure XX”.

Linking to documentation

✎🔗

Establishing names of headers

✎🔗

You give IDs to headers using the format:

### header title {#topic ID}

For example, for this subsection, we have used:

### Establishing names of headers {#establishing}

With this, we have given this header the ID “establishing”.

Special paragraphs and environments

✎🔗

Special paragraphs tags

✎🔗

The system supports parsing of some special paragraphs.

some of these might be redundant and will be eliminated. For now, I am documenting what is implemented.

Special paragraphs must be separated by a line

✎🔗

A special paragraph is marked by a special prefix. The list of special prefixes is given in the next section.

There must be an empty line before a special paragraph; this is because in Markdown a paragraph starts only after an empty line.

This is checked automatically, and the compilation will abort if the mistake is found.

For example, this is invalid:

See: this book
See: this other book

This is correct:

See: this book

See: this other book

Similarly, this is invalid:

Author: author
Maintainer: maintainer

and this is correct:

Author: author

Maintainer: maintainer

Other div environments

✎🔗

For these, note the rules:

  • You must include markdown="1".
  • There must be an empty line after the first div and before the closing /div.

Notes and questions

✎🔗

There are three environments: “comment”, “question”, “doubt”, that result in boxes that can be expanded by the user.

These are the one-paragraph forms:

Comment: this is a comment on one paragraph.
comment

this is a comment on one paragraph.

Question: this is a question on one paragraph.
question

this is a question on one paragraph.

Doubt: I have my doubts on one paragraph.
doubt

I have my doubts on one paragraph.

These are the multiple-paragraphs forms:

<div class='comment' markdown='1'>
A comment...

A second paragraph...
</div>
comment

A comment…

A second paragraph…

<div class='question' markdown='1'>
A question...

A second paragraph...
</div>
question

A question…

A second paragraph…

<div class='doubt' markdown='1'>
A question...

Should it not be:

    $ alternative command

A second paragraph...
</div>
doubt

A question…

Should it not be:

$ alternative command

A second paragraph…

Because of mathjax bug

Using LaTeX constructs in documentation

✎🔗

Working knowledge of LaTeX.

LaTeX equations

✎🔗

We can refer to equations, such as \eqref{eq:one}:

\begin{equation} 2a = a + a \label{eq:one}\tag{1} \end{equation}

This uses align and contains \eqref{eq:two} and \eqref{eq:three}.

\begin{align} a &= b \label{eq:two}\tag{2} \\ &= c \label{eq:three}\tag{3} \end{align}

We can refer to equations, such as \eqref{eq:one}:

\begin{equation}
    2a = a + a          \label{eq:one}
\end{equation}

This uses `align` and contains  \eqref{eq:two} and \eqref{eq:three}.

\begin{align}
    a &= b       \label{eq:two} \\
      &= c       \label{eq:three}
\end{align}

Note that referring to the equations is done using the syntax \eqref{eq:name}, rather than [](#eq:name).

Bibliography support

✎🔗

You need to have installed bibtex2html.

The system supports Bibtex files.

Place *.bib files anywhere in the directory.

Then you can refer to them using the syntax:

[](#bib:bibtex ID)

For example:

Please see [](#bib:siciliano07handbook).

Will result in:

Please see [3].

Embedding Latex in Figures through SVG

✎🔗

In order to compile the figures into PDFs you need to have Inkscape installed. Instructions to download and install Inkscape are here.

To embed latex in your figures, you can add it directly to a file and save it as filename.svg file and save anywhere in the /docs directory.

You can run:

$ make process-svg-figs

And the SVG file will be compiled into a PDF figure with the LaTeX commands properly interpreted.

You can then include the PDF file in a normal way (Section 2.5 - Figures) using filename.pdf as the filename in the <img> tag.

Image saved as svg
Image as PDF after processing
Embedding LaTeX in images

It can take a bit of work to get the positioning of the code to appear properly on the figure.

Because of mathjax bug

Advanced Markduck guide

✎🔗

Embedding videos

✎🔗

It is possible to embed Vimeo videos in the documentation.

Do not upload the videos to your personal Vimeo account; they must all be posted to the Duckietown Engineering account.

This is the syntax:

<dtvideo src="vimeo:vimeo ID"/>

For example, this code:

<div figure-id="fig:example-embed">
    <figcaption>Cool Duckietown by night</figcaption>
    <dtvideo src="vimeo:152825632"/>
</div>

produces this result:

The video is at https://vimeo.com/152825632.

Cool Duckietown by night

Depending on the output media, the result will change:

  • On the online book, the result is that a player is embedded.
  • On the e-book version, the result is that a thumbnail is produced, with a link to the video;
  • On the dead-tree version, a thumbnail is produced with a QR code linking to the video (TODO).

move-here tag

✎🔗

If a file contains the tag move-here, the fragment pointed by the src attribute is moved at the place of the tag.

This is used for autogenerated documentation.

Syntax:

# Node `node`

<move-here src='#package-node-autogenerated'/>

*Compiling the PDF version

✎🔗

This part describes how to compile the PDF version.

The dependencies below are harder to install. If you don’t manage to do it, then you only lose the ability to compile the PDF. You can do make compile to compile the HTML version, but you cannot do make compile-pdf.

Markduck troubleshooting

✎🔗

Common mistakes with Markdown

✎🔗

Here are some common mistakes encountered.

Not properly starting a list

✎🔗

There must be an empty line before the list starts.

This is correct:

I want to learn:

- robotics
- computer vision
- underwater basket weaving

This is incorrect:

I want to learn:
- robotics
- computer vision
- underwater basket weaving

and it will be rendered as follows:

I want to learn: - robotics - computer vision - underwater basket weaving

Because of mathjax bug

The Duckuments bot

✎🔗

This is an advanced section mainly for Liam.

Documentation style guide

✎🔗

This chapter describes the conventions for writing the technical documentation.

Learning in Duckietown

✎🔗

Jacopo

This chapter is a draft.

We do not refer to teachers and students

We refer to learners. Instructors are learners ahead in the learning curve in comparison to other learners

We don’t examinate

We assess competences

The learning feedback loop

✎🔗

A relevant contribution of modern educational theory is recognizing the importance of feedback in the learning process ().

We love feedback, as it stands at the foundation of control systems theory.

The learning loop.

Here, we provide definitions of the key elements in a learning feedback loop, and analogies to their control systems counterparts.

Intended Learning Outcomes

✎🔗

Intended Learning Outcome
An intended learning outcome is a desired, measurable, output of the learning process.

Intended learning outcomes are:

  • the starting point in the construction of a learning activity ([1]),

  • more effective when formulated with active verbs,

  • the equivalent of reference trajectories, or setpoints, in control systems. They represent the ideal output of the controlled system.

Students will understand robotics

Students each build a Duckiebot, implement software and produce demos

Learning Activities

✎🔗

Learning Activities
Methods chosen by the instructor to ease the learners achievement of the intended learning outcomes.

Active learning practices ([1]) have been shown to improve learning.

Instructor explains at the blackboard

Learners work in groups to achieve objectives

Learning activities are analogous to the the output of the controller (instructor), or input to the system (learners). They have to meet the requirements imposed by external constraints, not shown in

Assessment

✎🔗

Assessment
An assessment is a procedure to quantify the level of fulfillment of learning outcomes.

An assessment is analogous to a sensor in a control system loop. It measures the system’s output.

Examples of assessments include: quizzes, colloquia, produced documentation, homework, etc.

Knowledge, Skills and Competences

✎🔗

Here, we provide definitions for knowledge, skills and competences, in addition to describing their relationships (Figure 10.2).

The relationship between Knowledge, Skills and Competences.

Knowledge

✎🔗

Knowledge
Theoretical facts and information aimed at enabling understanding, and generating or improving skills.

Bayesian inference is handy piece of knowledge when doing robotics.

Practice

✎🔗

Practice
Practical procedures aimed at generating or improving skills, either directly or indirectly by improving knowledge.

Exercises and proofs can be used to practice different skills.

Skills

✎🔗

Skills
A proficiency, facility or dexterity that enables carrying out a function. Skills stem from knowledge, practice and/or aptitude. Skills can be clustered in cognitive, technical and interpersonal, respectively relating to ideas, things and people.

Analyzing tradeoffs between performances and constraints is a critical cognitive skill for robotics.

Python language is a useful technical skill in robotics.

Public speaking is a valuable interpersonal skill useful beyond robotics.

In Duckietown we formalize didactic indivisible units, or atoms, aimed at improving skills through knowledge and practice. Knowledge atoms are listed in XXX. We define as practice atoms:

add general reference to all learning atmos, folder atoms_30_learning_material

Exercise
An exercise is a practice atom aimed at improving technical skills. Exercises are listed in XXX.

Exercises are targeted to different “things” to which technical skills are related. They may be mathematical exercises aimed at practicing a method, or they may be coding exercises aimed at practicing resolutions of hardware implementation challenges.

Proof
A proof is a practice atom aimed at improving cognitive skills.

Deriving the Kalman filter equations helps practice the idea that there is no better approach to state estimation for linear time invariant systems, with “well behaved” measurement and process noises.

Competences

✎🔗

Competences
Set of skills and/or knowledge that leads to superior performance in carrying out a function. Competences must be measurable.

Competences are desirable intended learning outcomes, and typically address the how of the learning process.

Programming is a competence. It requires a skill, e.g., Python, and knowledge, e.g., Bayesian inference, to know what to code. Practice can help improve knowledge or hone skills.

Because of mathjax bug

Knowledge graph

✎🔗

This chapter describes something that is not implemented yet.

Markdown format for text-like atoms

✎🔗

For the text-like resources, they are described in Markdown files.

The name of the file does not matter.

All files are encoded in UTF-8.

Each file starts with a H1 header. The contents is the title.

The header has the following attributes:

  1. The ID. ({#ID})
  2. The status is given by an attribute status, which should be value of the values in Table 11.1.
  3. (Optional) The language is given by an attribute lang ({lang=en-US}).
  4. (Optional) The type is given by an attribute type ({type=demo}).

Here is an example of a header with all the attributes:

# Odometry calibration {#odometry-calibration lang=en-US type='text/theory' status=ready}

This first paragraph will be used as the "summary" for this text.
calibration.en.md

And this is how the Italian translation would look like:

# Calibrazione dell'odometria {#odometry-calibration lang=it type='text/theory' status=draft}

Questo paragrafo sarà usato come un sommario del testo.
calibration.it.md

Translations

✎🔗

This part is not implemented yet.

Operation manual - Duckiebot

✎🔗

In this section you will find information to obtain the necessary equipment for Duckietowns and different Duckiebot configurations.

Because of mathjax bug

Duckiebot configurations

✎🔗

nothing

Knowledge of Duckiebot configuration naming conventions, their components and functionalities.

After reviewing the configurations, you can proceed to purchasing the components, reading a description of the components, or assembling your chosen configuration.

We define different Duckiebot configurations depending on their time of use and hardware components. This is a good starting point if you are wondering what parts you should obtain to get started.

Branch configuration releases: Fall 2017

✎🔗

All branches release their hardware in two phases, namely a and b.

Soldering boards (DB17)

✎🔗

Shiying

Parts: Duckiebot DB17 parts. The acquisition process is explained in Unit C-2 - Acquiring the parts (DB17-jwd). The configurations are described in Unit C-1 - Duckiebot configurations. In particular:

Tools: Solderer

Experience: some novice-level experience with soldering.

Time: 30 minutes

Soldered DC Motor HAT

It is better to be safe than sorry. Soldering is a potentially hazardous activity. There is a fire hazard as well as the risk of inhaling toxic fumes. Stop a second and make sure you are addressing the safety standards for soldering when following these instructions. If you have never soldered before, seek advice.

Preparing the power cable (DB17)

✎🔗

In configuration DB17 we will need a cable to power the DC motor HAT from the battery. The keen observer might have noticed that such a cable was not included in the DB17 Duckiebot parts chapter. Here, we create this cable by splitting open any USB-A cable, identifying and stripping the power wires, and using them to power the DC motor HAT. If you are unsure about the definitions of the different Duckiebot configurations, read Unit C-1 - Duckiebot configurations.

It is important to note that these instructions are relevant only for assembling a DB17-wjdc configuration Duckiebot (or any subset of it). If you intend to build a DB17-l configuration Duckiebot, you can skip these instructions.

One male USB-A to anything cable.

A pair of scissors.

A multimeter (only if you are not purchasing the suggested components)

Time: 5 minutes

One male USB-A to wires power cable

Assembling the Duckiebot (DB17-jwd)

✎🔗

Shiying Li

Once you have received the parts and soldered the necessary components, it is time to assemble them in a Duckiebot. Here, we provide the assembly instructions for configurations DB17-wjd.

Duckiebot DB17-wjd parts. The acquisition process is explained in Unit C-2 - Acquiring the parts (DB17-jwd).

Having soldered the DB17-wjd parts. The soldering process is explained in Unit C-3 - Soldering boards (DB17).

Having prepared the power cable. The power cable preparation is explained in Unit C-4 - Preparing the power cable (DB17). Note: Not necessary if you intend to build a DB17-l configuration.

Having installed the image on the MicroSD card. The instructions on how to reproduce the Duckiebot system image are in Unit C-7 - Reproducing the image.

Time: about 40 minutes.

An assembled Duckiebot in configuration DB17-wjd.

The FAQ section at the bottom of this page may already answer some of you comments, questions or doubts.

While assembling the Duckiebot, try to make as symmetric (along the longitudinal axis) as you can. It will help going forward.

FAQ

✎🔗

If we have the bumpers, at what point should we add them?

You shouldn’t have the bumpers at this point. The function of bumpers is to keep the LEDs in place, i.e., they belong to DB17-l configuration. These instructions cover the DB17-jwd configurations. You will find the bumper assembly instructions in Unit E-3 - Assembling the Duckiebot (DB17-lc).

Yeah but I still have the bumpers and am reading this page. So?

The bumpers can be added after the Duckiebot assembly is complete.

I found it hard to mount the camera (the holes weren’t lining up).

Sometimes in life you have to push a little to make things happen. (But don’t push too much or things will break!)

The long camera cable is a bit annoying - I folded it and shoved it in between two hats.

The shorter cable is even more annoying. We suggest wrapping the long camera cable between the chassis and the Raspberry Pi. With some strategic planning, you can use the zipties that keep the battery in place to hold the camera cable in place as well (see figure below-to add)

add pretty cable handling pic

I found that the screwdriver that comes with the chassis kit is too fat to screw in the wires on the hat.

It is possible you got one of the fatter screwdrivers. You will need to figure it out yourself (or ask a TA for help).

I need something to cut the end of the zip tie with.

Scissors typically work out for these kind of jobs (and no, they’re not provided in a Fall 2017 Duckiebox).

Because of mathjax bug

Assembling the Duckiebot (DB17-wjd TTIC)

✎🔗

Andrea F. Daniele

Once you have received the parts and soldered the necessary components, it is time to assemble them in a Duckiebot. Here, we provide the assembly instructions for the configuration DB17-wjd (TTIC only).

Duckiebot DB17-wjd parts. The acquisition process is explained in Unit C-2 - Acquiring the parts (DB17-jwd).

Having soldered the DB17-wjd parts. The soldering process is explained in Unit C-3 - Soldering boards (DB17).

Having prepared the power cable. The power cable preparation is explained in Unit C-4 - Preparing the power cable (DB17). Note: Not necessary if you intend to build a DB17-l configuration.

Time: about 30 minutes.

An assembled Duckiebot in configuration DB17-wjd.

The FAQ section at the bottom of this page may already answer some of you comments, questions or doubts.

This section is comprised of 14 parts. Each part builds upon some of the previous parts, so make sure to follow them in the following order.

Motors

✎🔗

Open the Magician Chassis package (Figure 5.1) and take out the following components:

  • Chassis-bottom (1x)
  • DC Motors (2x)
  • Motor holders (4x)
  • M3x30 screw (4x)
  • M3 nuts (4x)

Figure 6.1 shows the components needed to complete this part of the tutorial.

Components needed to mount the motors.

Wheels

✎🔗

From the Magician Chassis package take the following components:

  • Wheels (2x)

Figure 6.7 shows the components needed to complete this part of the tutorial.

The wheels.

Omni-directional wheel

✎🔗

The Duckiebot is driven by controlling the wheels attached to the DC motors. Still, it requires a passive support on the back. In this configuration an omni-directional wheel is attached to the bottom plate of the chassis to provide such support.

From the Magician Chassis package take the following components:

  • Steel omni-directional wheel (1x)
  • Long metal spacers (2x)
  • M3x6 screws (4x)

Figure 6.10 shows the components needed to complete this part of the tutorial.

The omni-directional wheel with *2* long spacers and *4* M3x6 screws.

Chassis standoffs

✎🔗

From the Magician Chassis package take the following components:

  • Long metal spacers/standoffs (4x)
  • M3x6 screws (4x)

From the Duckiebot kit take the following components:

  • M3x5 nylon spacers/standoffs (4x)

Figure 6.14 shows the components needed to complete this part of the tutorial.

The standoffs to mount on the bottom plate.

Camera kit

✎🔗

From the Magician Chassis package take the following components:

  • M3x10 flathead screws (2x)
  • M3 nuts (2x)

From the Duckiebot kit take the following components:

  • Camera Module (1x)
  • (Optional) 300mm Camera cable (1x)
  • Camera mount (1x)

If you have camera cables of different lengths available, keep in mind that both are going to work. We suggest to use the longer one, and wrap the extra length under the Raspberry Pi stack.

Figure 6.18 shows the components needed to complete this part of the tutorial.

The parts needed to fix the camera on the top plate.

Heat sinks

✎🔗

From the Duckiebot kit take the following components:

  • Raspberry Pi 3 (1x)
  • Heat sinks (2x)
  • Camera mount (1x)

Figure 6.24 shows the components needed to complete this part of the tutorial.

The heat sinks and the Raspberry Pi 3.

Raspberry Pi 3

✎🔗

From the Magician Chassis package take the following components:

  • Top plate (with camera attached) (1x)

From the Duckiebot kit take the following components:

  • Raspberry Pi 3 (with heat sinks) (1x)
  • M2.5x12 nylon spacers/standoffs (8x)
  • M2.5 nylon hex nuts (4x)

Figure 6.27 shows the components needed to complete this part of the tutorial.

The parts needed to mount the Raspberry Pi 3 on the top plate.

Top plate

✎🔗

From the Magician Chassis package take the following components:

  • Bottom plate (with motors, wheels and standoffs attached) (1x)
  • Top plate (with camera and Raspberry Pi 3 attached) (1x)
  • M3x6 screws (4x)

Figure 6.32 shows the components needed to complete this part of the tutorial.

The parts needed to secure the top plate to the bottom plate.

USB Power cable

✎🔗

The power cable preparation is explained in Unit C-4 - Preparing the power cable (DB17).

DC Stepper Motor HAT

✎🔗

From the Duckiebot kit take the following components:

Figure 6.36 shows the components needed to complete this part of the tutorial.

The parts needed to add the DC Stepper Motor HAT to the Duckiebot.

Battery

✎🔗

From the Duckiebot kit take the following components:

  • Battery (1x)
  • Zip tie (1x)
  • Short micro USB cable (1x)

Figure 6.43 shows the components needed to complete this part of the tutorial.

The parts needed to add the battery to the Duckiebot.

Upgrade to DB17-w

✎🔗

This upgrade equips the Duckiebot with a secondary, faster, Wi-Fi connection, ideal for image streaming. The new configuration is called DB17-w.

Figure 6.46 shows the components needed to complete this upgrade.

The parts needed to upgrade the Duckiebot to the configuration DB17-w.

Upgrade to DB17-j

✎🔗

This upgrade equips the Duckiebot with manual remote control capabilities. It is particularly useful for getting the Duckiebot out of tight spots or letting younger ones have a drive, in addition to providing handy shortcuts to different functions in development phase. The new configuration is called DB17-j.

Figure 6.48 shows the components needed to complete this upgrade.

The parts needed to upgrade the Duckiebot to the configuration DB17-j.

The joystick comes with a USB receiver (as shown in Figure 6.48).

Upgrade to DB17-d

✎🔗

This upgrade equips the Duckiebot with an external hard drive that is convenient for storing videos (logs) as it provides both extra capacity and faster data transfer rates than the microSD card in the Raspberry Pi 3. Moreover, it is easy to unplug it from the Duckiebot at the end of the day and bring it over to a computer for downloading and analyzing stored data. The new configuration is called DB17-d.

Figure 6.50 shows the components needed to complete this upgrade.

The parts needed to upgrade the Duckiebot to the configuration DB17-d.

FAQ

✎🔗

If we have the bumpers, at what point should we add them?

You shouldn’t have the bumpers at this point. The function of the bumpers is to keep the LEDs in place, i.e., they belong to DB17-l configuration. These instructions cover the DB17-wjd configurations. You will find the bumper assembly instructions in Unit E-3 - Assembling the Duckiebot (DB17-lc).

Yeah but I still have the bumpers and am reading this page. So?

The bumpers can be added after the Duckiebot assembly is complete.

I found it hard to mount the camera (the holes weren’t lining up).

Sometimes in life you have to push a little to make things happen. (But don’t push too much or things will break!)

The long camera cable is a bit annoying - I folded it and shoved it in between two hats.

The shorter cable is even more annoying. We suggest wrapping the long camera cable between the chassis and the Raspberry Pi. With some strategic planning, you can use the zipties that keep the battery in place to hold the camera cable in place as well (see figure below-to add)

add pretty cable handling pic

I found that the screwdriver that comes with the chassis kit is too fat to screw in the wires on the hat.

It is possible you got one of the fatter screwdrivers. You will need to figure it out yourself (or ask a TA for help).

I need something to cut the end of the zip tie with.

Scissors typically work out for these kind of jobs (and no, they’re not provided in a Fall 2017 Duckiebox).

Because of mathjax bug

Reproducing the image

✎🔗

These are the instructions to reproduce the Ubuntu image that we use.

Please note that the image is already available, so you don’t need to do this.

However, this documentation is useful if you would like to port the software to a different distribution. Also, we periodically run through these instructions to make sure that they work.

Just in case, let’s re-state this: in Fall 2017, you don’t necessarily need to do the following.

Internet connection to download the packages.

A PC running any Linux with an SD card reader.

Time: about 4 hours (most of it spent waiting for things to download/compile).

A baseline Ubuntu Mate 16.04.2 image with updated software.

Passwordless sudo

✎🔗

First, make vi the default editor, using

$ sudo update-alternatives --config editor

and then choose vim.basic.

Then run:

$ sudo visudo

And then change this line:

%sudo   ALL=(ALL:ALL) ALL

into this line:

%sudo   ALL=(ALL:ALL) NOPASSWD:ALL

Installing Ubuntu on laptops

✎🔗

Andrea

Before you prepare the Duckiebot, you need to have a laptop with Ubuntu installed.

A laptop with free disk space.

Internet connection to download the Ubuntu image.

About 30 minutes.

A laptop ready to be used for Duckietown.

Duckiebot Initialization

✎🔗

Andrea

An SD card of dimensions at least 16 GB.

A computer with an internet connection, an SD card reader, and 16 GB of free space.

An assembled Duckiebot in configuration DB17. This is the result of Unit C-5 - Assembling the Duckiebot (DB17-jwd).

A Duckiebot that is configured correctly, that you can connect to with your laptop and hopefully also has internet access

Acquire and burn the image

✎🔗

On the laptop, download the compressed image at this URL:

https://www.dropbox.com/s/ckpqpp0cav3aucb/duckiebot-RPI3-AD-2017-09-12.img.xz?dl=1

The size is 1.7 GB.

You can use:

$ wget -O duckiebot-RPI3-AD-2017-09-12.img.xz URL above
comment

The original was:

$ curl -o duckiebot-RPI3-AD-2017-09-12.img.xz URL above

It looks like that curl cannot be used with Drobpox links because it does not follow redirects.

To make sure that the image is downloaded correctly, compute its hash using the program sha256sum:

$ sha256sum duckiebot-RPI3-AD-2017-09-12.img.xz
7136f9049b230de68e8b2d6df29ece844a3f830cc96014aaa92c6d3f247b6130  duckiebot-RPI3-AD-2017-09-12.img.xz

Compare the hash that you obtain with the hash above. If they are different, there was some problem in downloading the image.

Uncompress the file:

$ xz -d -k --verbose duckiebot-RPI3-AD-2017-09-12.img.xz

This will create a file of 11 GB in size.

Next, burn the image on disk.

The procedure of how to burn an image is explained in Section 17.2 - How to burn an image to an SD card.

Expand your filesystem

✎🔗

If your SD card is larger than the image, you’ll want to expand the filesystem on your robot so that you can use all of the space available. Achieve this with:

duckiebot $ sudo raspi-config --expand-rootfs

and then reboot

duckiebot $ sudo shutdown -r now

once rebooted you can test whether this was successful by doing

duckiebot $ df -lh

the output should give you something like:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  6.3G  8.2G  44% /
devtmpfs        303M     0  303M   0% /dev
tmpfs           431M     0  431M   0% /dev/shm
tmpfs           431M   12M  420M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           431M     0  431M   0% /sys/fs/cgroup
/dev/mmcblk0p1   63M   21M   43M  34% /boot
tmpfs            87M     0   87M   0% /run/user/1000

You should see that the Size of your /dev/root Filesystem is “close” to the size of your SD card.

Create your user

✎🔗

You must not use the ubuntu user for development. Instead, you need to create a new user.

Choose a user name, which we will refer to as username.

To create a new user:

duckiebot $ sudo useradd -m username

Make the user an administrator by adding it to the group sudo:

duckiebot $ sudo adduser username sudo

Make the user a member of the groups input, video, and i2c

duckiebot $ sudo adduser username input
duckiebot $ sudo adduser username video
duckiebot $ sudo adduser username i2c

Set the shell bash:

duckiebot $ sudo chsh -s /bin/bash username

To set a password, use:

duckiebot $ sudo passwd username

At this point, you should be able to login to the new user from the laptop using the password:

laptop $ ssh username@robot name

Next, you should repeat some steps that we already described.

comment

What steps?? -LP

Add SSH alias

✎🔗

Once you have your SSH key pair on both your laptop and your Duckiebot, as well as your new user- and hostname set up on your Duckiebot, then you should set up an SSH alias as described in Section 15.1 - SSH aliases. This allows your to log in for example with

laptop $ ssh abc

instead of

laptop $ ssh username@robot name

where you can chose abc to be any alias / shortcut.

Hardware check: camera

✎🔗

Check that the camera is connected using this command:

duckiebot $ vcgencmd get_camera
supported=1 detected=1

If you see detected=0, it means that the hardware connection is not working.

You can test the camera right away using a command-line utility called raspistill.

Use the raspistill command to capture the file out.jpg:

duckiebot $ raspistill -t 1 -o out.jpg

Then download out.jpg to your computer using scp for inspection.

For instructions on how to use scp, see Subsection 21.1.1 - Download a file with SCP.

✎🔗

In order to show that your Duckiebot is ready for the task of driving around happy little duckies, the robot has to fly the Duckietown flag. When you are still logged in to the Duckiebot you can download and install the banner like this:

Download the ANSI art file from Github:

duckiebot $ wget --no-check-certificate -O duckie.art "https://raw.githubusercontent.com/duckietown/Software/master/misc/duckie.art"

(optional) If you want, you can preview the logo by just outputting it onto the command line:

duckiebot $ cat duckie.art

Next up create a new empty text file in your favorite editor and add the code for showing your duckie pride:

Let’s say I use nano, I open a new file:

duckiebot $ nano 20-duckie

And in there I add the following code (which by itself just prints the duckie logo):

#!/bin/sh
printf "\n$(cat /etc/update-motd.d/duckie.art)\n"

Then save and close the file. Finally you have to make this file executable…

duckiebot $ chmod +x 20-duckie

…and copy both the duckie logo and the script into a specific directory /etc/update-motd.d to make it appear when you login via SSH. motd stands for “message of the day”. This is a mechanism for system administrators to show users news and messages when they login. Every executable script in this directory which has a filename a la NN-some name will get exected when a user logs in, where NN is a two digit number that indicates the order.

sudo cp duckie.art /etc/update-motd.d
sudo cp 20-duckie /etc/update-motd.d

Finally log out of SSH via exit and log back in to see duckie goodness.

Networking aka the hardest part

✎🔗

A Duckiebot in configuration DB17-CO+w

Either a router that you have control over that has internet access, or your credentials for connecting to an existing wireless network

Patience (channel your inner Yoda)

A Duckiebot that you can connect to and that is connected to the internet

this page is primarily for folks operating with the “two-network” configuration, C0+w. For a one adapter setup you will can skip directly to Section 10.2 - Setting up wireless network configuration, but you will have to connect to a network that you can ssh through.

The basic idea is that we are going to use the “Edimax” thumbdrive adapter to create a dedicated wireless network that you can always connect to with your laptop. Then we are going to use the built-in Broadcom chip on the Pi to connect to the internet, and then the network will be bridged.

Setting up wireless network configuration

✎🔗

You are connected to the Duckiebot via WiFi, but the Duckiebot also needs to connect to the internet in order to get updates and install some software. This part is a little bit more of a “black art” since we cannot predict every possible network configurations. Below are some settings that have been verified to work in different situations:

Option 5: ETH Wifi

✎🔗

The following instructions will lead you to connect your PI to the “eth” wifi network.

First, run the following on duckiebot

duckiebot $ iwconfig
...

lo        no wireless extensions.

enxbxxxxxxxxxxx  no wireless extensions.

...

Make note of the name enxbxxxxxxxxxxx. xxxxxxxxxxx should be a string that has 11 characters that is formed by numbers and lower case letters.

Second, edit the file /etc/network/interfaces which requires sudo so that it looks like the following, and make sure the enxbxxxxxxxxxxx matches.

Pay special attention on the line “pre-up wpa_supplicant -B -D wext -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf”.This is expected to be exactly one line instead of two but due to formatting issue it is shown as two lines.

Also, make sure every characters match exactly with the provided ones. TAs will not help you to do spelling error check. 

# interfaces(5) file used by ifup(8) and ifdown(8) Include files from /etc/network/     interfaces.d:
source-directory /etc/network/interfaces.d

# The loopback network interface
auto lo
auto enxbxxxxxxxxxxx

# the wired network setting 
iface enxbxxxxxxxxxxx inet dhcp

# the wireless network setting
auto wlan0
allow-hotplug wlan0
iface wlan0 inet dhcp
    pre-up wpa_supplicant -B -D wext -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
    post-down killall -q wpa_supplicant

Third, edit the file /etc/wpa_supplicant/wpa_supplicant.conf which requires sudo so that it looks like the following, and make sure you substitute [identity] and [password] content with your eth account information: 

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="eth"
    key_mgmt=WPA-EAP
    group=CCMP TKIP
    pairwise=CCMP TKIP
    eap=PEAP
    proto=RSN
    identity="your user name goes here"
    password="your password goes here"
    phase1="peaplabel=0"
    phase2="auth=MSCHAPV2"
    priority=1
}

Fourth, reboot your PI. 

duckiebot $ sudo reboot

Then everything shall be fine. The PI will connect to “eth” automatically everytime it starts.

Note that, if something went wrong, your Duckiebot tries to connect to the network for 5.5mins at startup while it’s blocking SSH connection to it completely (“Connection refused” error when connecting). If this is the case, please wait those 5.5mins until your Duckiebot lets you connect again and recheck your settings.

Find a solution to this since it occurs very often

Because of mathjax bug

Software setup and RC remote control

✎🔗

Laptop configured, according to Unit C-8 - Installing Ubuntu on laptops.

You have configured the Duckiebot. The procedure is documented in Unit C-9 - Duckiebot Initialization.

You have created a Github account and configured public keys, both for the laptop and for the Duckiebot. The procedure is documented in Unit J-29 - Setup Github access.

You can run the joystick demo.

Clone the Duckietown repository

✎🔗

Clone the repository in the directory ~/duckietown:

duckiebot $ git clone git@github.com:duckietown/Software.git ~/duckietown

For the above to succeed you should have a Github account already set up.

It should not ask for a password.

you must not clone the repository using the URL starting with https. Later steps will fail.

Set up the ROS environment on the Duckiebot

✎🔗

All the following commands should be run in the ~/duckietown directory:

duckiebot $ cd ~/duckietown

Now we are ready to make the workspace. First you need to source the baseline ROS environment:

duckiebot $ source /opt/ros/kinetic/setup.bash

Then, build the workspace using:

duckiebot $ catkin_make -C catkin_ws/

For more information about catkin_make, see Section 30.6 - catkin_make.

there is a known bug, for which it fails the first time on the Raspberry Pi. Try again; it will work.

comment

I got no error on first execution on the Raspberry Pi

Clone the duckiefleet repository

✎🔗

Clone the relevant duckiefleet repository into ~/duckiefleet.

See Subsection 4.1.2 - Duckiefleet directory DUCKIEFLEET_ROOT to find the right duckiefleet repository.

In ~/.bashrc set DUCKIEFLEET_ROOT to point to the directory:

export DUCKIEFLEET_ROOT=~/duckiefleet

Also, make sure that you execute ~/.bashrc in the current shell by running the command:

source ~/.bashrc

Add your vehicle data to the robot database

✎🔗

Next, you need to add your robot to the vehicles database. This is not optional and required in order to launch any ROS scripts.

You have already a copy of the vehicles database in the folder robots of DUCKIEFLEET_ROOT.

Copy the file emma.robot.yaml to robotname.robot.yaml, where robotname is your robot’s hostname. Then edit the copied file to represent your Duckiebot.

For information about the format, see Section 4.2 - The “scuderia” (vehicle database).

Generate the machines file.

The procedure is listed here: Section 4.3 - The machines file.

Finally, push your robot configuration to the duckiefleet repo.

Test that the joystick is detected

✎🔗

Plug the joystick receiver in one of the USB port on the Raspberry Pi.

To make sure that the joystick is detected, run:

duckiebot $ ls /dev/input/

and check if there is a device called js0 on the list.

Make sure that your user is in the group input and i2c:

duckiebot $ groups
username sudo input i2c

If input and i2c are not in the list, you missed a step. Ohi ohi! You are not following the instructions carefully!

Consult again Section 9.11 - Create your user.

To test whether or not the joystick itself is working properly, run:

duckiebot $ jstest /dev/input/js0

Move the joysticks and push the buttons. You should see the data displayed change according to your actions.

Reading from the camera

✎🔗

You have configured the Duckiebot. The procedure is documented in Unit C-9 - Duckiebot Initialization.

You know the basics of ROS (launch files, roslaunch, topics, rostopic).

put reference

You know that the camera works under ROS.

RC control launched remotely

✎🔗

Andrea

You can run the joystick demo from the Raspberry Pi. The procedure is documented in Unit C-11 - Software setup and RC remote control.

You can run the joystick demo from your laptop.

Rebuild the machines files

✎🔗

In a previous step you have created a robot configuration file and pushed it to the duckiefleet repo. Now you have to pull duckiefleet on the laptop and rebuild the machines configuration file there.

The procedure is documented in Section 11.5 - Add your vehicle data to the robot database.

Start the demo

✎🔗

Now you are ready to launch the joystick demo remotely.

Make sure that you can login with SSH without a password. From the laptop, run:

laptop $ ssh username@robot name.local

If this doesn’t work, you missed some previous steps.

Run this on the laptop:

laptop $ source environment.sh
laptop $ roslaunch duckietown joystick.launch veh:=robot name

You should be able to drive the vehicle with joystick just like the last example. Note that remotely launching nodes from your laptop doesn’t mean that the nodes are running on your laptop. They are still running on the Raspberry Pi in this case.

For more information about roslaunch, see Section 30.3 - roslaunch.

RC+camera remotely

✎🔗

Andrea

You can run the joystick demo remotely. The procedure is documented in Unit C-13 - RC control launched remotely.

You can read the camera data from ROS. The procedure is documented in Unit C-12 - Reading from the camera.

You know how to get around in Byobu. You can find the Byobu tutorial in Unit J-26 - Byobu.

You can run the joystick demo from your laptop and see the camera image on the laptop.

Interlude: Ergonomics

✎🔗

Andrea

So far, we have been spelling out all commands for you, to make sure that you understand what is going on.

Now, we will tell you about some shortcuts that you can use to save some time.

in the future you will have to debug problems, and these problems might be harder to understand if you rely blindly on the shortcuts.

Time: 5 minutes.

You will know about some useful shortcuts.

SSH aliases

✎🔗

Instead of using

$ ssh username@robot name.local

You can set up SSH so that you can use:

$ ssh my-robot

To do this, create a host section in ~/.ssh/config on your laptop with the following contents:

Host my-robot
    User username
    Hostname robot name.local

Here, you can choose any other string in place of “my-robot”.

Note that you cannot do

$ ping my-robot

You haven’t created another hostname, just an alias for SSH.

However, you can use the alias with all the tools that rely on SSH, including rsync and scp.

Wheel calibration

✎🔗

Andrea Daniele

You can run the joystick demo remotely. The procedure is documented in Unit C-13 - RC control launched remotely.

Calibrate the wheels of the Duckiebot such that it goes in a straight line when you command it to. Set the maximum speed of the Duckiebot.

Camera calibration

✎🔗

You can see the camera image on the laptop. The procedure is documented in Unit C-14 - RC+camera remotely.

You have all the repositories (described in Unit M-7 - Git usage guide for Fall 2017) cloned properly and you have your environment variables set properly.

Calibration for the robot camera.

Extrinsic calibration

✎🔗

Setup

✎🔗

Arrange the Duckiebot and checkerboard according to Figure 17.5. Note that the axis of the wheels should be aligned with the y-axis (Figure 17.5).

Figure 17.6 shows a view of the calibration checkerboard from the Duckiebot. To ensure proper calibration there should be no clutter in the background and two A4 papers should be aligned next to each other.

Calibration procedure

✎🔗 ✎

Log in into your robot using SSH and launch the camera:

duckiebot $ cd duckietown root
duckiebot $ source environment.sh
duckiebot $ roslaunch duckietown camera.launch veh:=robot name raw:=true
✎

Run the ground_projection_node.py node on your laptop:

laptop $ cd duckietown root
laptop $ source environment.sh
laptop $ source set_ros_master.sh robot name
laptop $ roslaunch ground_projection ground_projection.launch  veh:=robot name local:=true
✎

Check that everything is working properly. In a new terminal (with environment sourced and ros_master set to point to your robot as above)

laptop $ rostopic list

You should see new ros topics:

/robot name/camera_node/camera_info
/robot name/camera_node/framerate_high_switch
/robot name/camera_node/image/compressed
/robot name/camera_node/image/raw
/robot name/camera_node/raw_camera_info

The ground_projection node has two services. They are not used during operation. They just provide a command line interface to trigger the extrinsic calibration (and for debugging).

laptop $ rosservice list

You should see something like this:

...
/robot name/ground_projection/estimate_homography
/robot name/ground_projection/get_ground_coordinate
...

If you want to check whether your camera output is similar to the one at the Figure 17.6 you can start rqt_image_view:

laptop $ rosrun rqt_image_view rqt_image_view

In the rqt_image_view interface, click on the drop-down list and choose the image topic:

/robot name/camera_node/image/compressed
✎

Now you can estimate the homography by executing the following command (in a new terminal):

laptop $ rosservice call /robot name/ground_projection/estimate_homography

This will do the extrinsic calibration and automatically save the file to your laptop:

duckiefleet root/calibrations/camera_extrinsic/robot name.yaml

As before, add this file to your local Git repository on your laptop, push the changes to your branch and do a pull request to master. Finally, you will want to update the local repository on your Duckiebot.

Updated camera calibration and validation

✎🔗

Here is an updated, more practical extrinsic calibration and validation procedure.

Place the robot on the pattern

✎🔗

Arrange the Duckiebot and checkerboard according to Figure 18.1. Note that the axis of the wheels should be aligned with the y-axis (Figure 18.1).

Figure 18.2 shows a view of the calibration checkerboard from the Duckiebot. To ensure proper calibration there should be no clutter in the background and two A4 papers should be aligned next to each other.

Extrinsic calibration procedure

✎🔗

Run the following on the Duckiebot:

duckiebot $ rosrun complete_image_pipeline calibrate_extrinsics

That’s it!

No laptop is required.

You can also look at the output files produced, to make sure it looks reasonable. It should look like Figure 18.3.

Note the difference between the two types of rectification:

  1. In bgr_rectified the rectified frame coordinates are chosen so that the frame is filled entirely. Note the image is stretched - the April tags are not square. This is the rectification used in the lane localization pipeline. It doesn’t matter that the image is stretched, because the homography learned will account for that deformation.

  2. In rectified_full_ratio_auto the image is not stretched. The camera matrix is preserved. This means that the aspect ratio is the same. In particular note the April tags are square. If you do something with April tags, you need this rectification.

Camera validation by simulation

✎🔗

You can run the following command to make sure that the camera calibration is reasonable:

duckiebot $ rosrun complete_image_pipeline validate_calibration

What this does is simulating what the robot should see, if the models were correct (Figure 18.4).

Result of validate_calibration.

Then it also tries to localize on the simulated data (Figure 18.5). It usual achieves impressive calibration results!

Simulations are doomed to succeed.

Output of validate_calibration: localization in simulated environment.

Camera validation by running one-shot localization

✎🔗

Place the robot in a lane.

Run the following command:

duckiebot $ rosrun complete_image_pipeline single_image_pipeline

What this does is taking one snapshot and performing localization on that single image. The output will be useful to check that everything is ok.

Taking and verifying a log

✎🔗

Unit C-12 - Reading from the camera

Unit C-11 - Software setup and RC remote control

A verified log.

Record the log

✎🔗

Verify a log

✎🔗

On the Duckiebot run:

duckiebot $ rosbag info FULL_PATH_TO_BAG --freq

Then:

  • verify that the “duration” of the log seems “reasonable” - it’s about as long as you ran the log command for

  • verify that the “size” of the log seems “reasonable” - the log size should grow at about 220MB/min

  • verify in the output that your camera was publishing very close to 30.0Hz and verify that you joysick was publishing at a rate between 3Hz and 6Hz.

More complex log verification methods.

Because of mathjax bug

Troubleshooting

✎🔗

The Raspberry Pi of the Duckiebot is connected to the battery.

The Stepper Motor HAT is connected to the battery.

You have a problem!

I cannot access my Duckiebot via SSH

✎🔗

When I run ssh robot_name.local I get the error ssh: Could not resolve hostname robot_name.local.

Make sure that your Duckiebot is ON. Connect it to a monitor, a mouse and a keyboard. Run the command

duckiebot $ sudo service avahi-daemon status

You should get something like the following

● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
   Loaded: loaded (/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2017-10-22 00:07:53 CEST; 1 day 3h ago
 Main PID: 699 (avahi-daemon)
   Status: "avahi-daemon 0.6.32-rc starting up."
   CGroup: /system.slice/avahi-daemon.service
           ├─699 avahi-daemon: running [robot_name_in_avahi.local
           └─727 avahi-daemon: chroot helpe

Avahi is the module that in Ubuntu implements the mDNS responder. The mDNS responder is responsible for advertising the hostname of the Duckiebot on the network so that everybody else within the same network can run the command ping robot_name.local and reach your Duckiebot. Focus on the line containing the hostname published by the avahi-daemon on the network (i.e., the line that contains robot_name_in_avahi.local). If robot_name_in_avahi matches the robot_name, go to the next Resolution point. If robot_name_in_avahi has the form robot_name-XX, where XX can be any number, modify the file /etc/avahi/avahi-daemon.conf as shown below.

Identify the line

use-ipv6=yes

and change it to

use-ipv6=no

Identify the line

#publish-aaaa-on-ipv4=yes

and change it to

publish-aaaa-on-ipv4=no

Restart Avahi by running the command

duckiebot $ sudo service avahi-daemon restart

The Duckiebot does not move

✎🔗

I can SSH into my Duckiebot and run the joystick demo but the joystick does not move the wheels.

Press the button on the side of the battery (Figure 20.1).

Check that the red indicator on the joystick stopped blinking.

Bad joystick status

Bad joystick status

The joystick is connected (as shown in Figure 20.2b - Bad joystick status) but the Duckiebot still does not move.

Make sure that the controller is connected to the Duckiebot and that the OS receives the data from it. Run

duckiebot $ jstest /dev/input/js0

If you receive the error

jstest: No such file or directory

it means that the USB receiver is not connected to the Raspberry Pi or is broken. If the command above shows something like the following

Driver version is 2.1.0.
Joystick (ShanWan PC/PS3/Android) has 8 axes (X, Y, Z, Rz, Gas, Brake, Hat0X, Hat0Y)
and 15 buttons (BtnX, BtnY, BtnZ, BtnTL, BtnTR, BtnTL2, BtnTR2, BtnSelect, BtnStart, BtnMode, BtnThumbL, BtnThumbR, ?, ?, ?).
Testing ... (interrupt to exit)
Axes:  0:     0  1:     0  2:     0  3:     0  4:-32767  5:-32767  6:     0  7:     0 Buttons:  0:off  1:off  2:off  3:off  4:off  5:off  6:off  7:off  8:off  9:off 10:off 11:off 12:off 13:off 14:off

it means that the USB receiver is connected to the Raspberry Pi. Leave the terminal above open and use the joystick to command the Duckiebot. If you observe that the numbers shown in the terminal change according to the commands sent through the joystick than the problem is in ROS. Make sure that the joystick demo is launched. Restart the Duckiebot if needed and try again.

If the numbers do not change while using the joystick then follow this guide at the next Resolution point.

The controller might be connected to another Duckiebot nearby. Turn off the controller, go to a room with no other Duckiebots around and turn the controller back on. Retry.

Because of mathjax bug

Operation manual - Duckietown

✎🔗
Because of mathjax bug

Duckietown parts

✎🔗

Duckietowns are the cities where Duckiebots drive. Here, we provide a link to all bits and pieces that are needed to build a Duckietown, along with their price tag. Note that while the topography of the map is highly customable, we recommend using the components listed below. Before purchasing components for a Duckietown, read Unit D-2 - Duckietown Appearance Specification to understand how Duckietowns are built.

In general, keep in mind that:

  • The links might expire, or the prices might vary.
  • Shipping times and fees vary, and are not included in the prices shown below.
  • Substitutions are probably not OK, unless you are OK in writing some software.

Cost (per $m^2$): USD ??? + Shipping Fees

Time: ??? days (average shipping time)

A kit of parts ready to be assembled in a Duckietown.

Assembling a Duckietown.

Figure out costs

Traffic lights Parts

✎🔗

Traffic lights regulate intersections in Duckietown. Here, we provide a link to all bits and pieces that are needed to build a traffic light, along with their price tag. You will need one traffic per either three, or four way intersections. The components listed below meet the appearence specifications described in Unit D-2 - Duckietown Appearance Specification.

In general, keep in mind that:

  • The links might expire, or the prices might vary.
  • Shipping times and fees vary, and are not included in the prices shown below.
  • Substitutions are probably OK, if you are willing to write some software.

Cost: USD ??? + Shipping Fees

Time: ??? days (average shipping time)

A kit of parts ready to be assembled in a traffic light.

Assembling a traffic light.

Estimate time and costs

Duckietown Appearance Specification

✎🔗

This document describes the Duckietown specification. These are a set of rules for which a functional system has been verified.

Any Duckietown not adhering to the rules described here cannot call itself a “Duckietown”, since it is not one.

Additionally, any Duckietown not adhering to these rules may cause the Duckiebots to fail in unexpected ways.

Traffic Signs

✎🔗

To print and assemble the signs refer to Unit D-3 - Signage.

Placement

✎🔗

Signs may appear on the opposite side and at the corner of the adjacent tile from which they are viewed. In the absence of any signs, it is assumed that all network flows are allowed so a sign MUST be placed and visible whenever this is not the case.

Signs must only be placed on empty tiles, or next to one of the other tile types if on the border of a map. The sign placements for four different cases are shown in Figure 2.8. At intersections, from each stop line 2 signs should be clearly visible: 1) the intersection type (traffic light or stop sign) and 2) the intersection topology.

At present, 4-way intersections must be equipped with traffic lights for safe navigation.

4-way intersection
3-way intersection
straight road
curved road
Placement of Traffic Signs

On straight and curved roads, additional signs can be added as desired. Their placement is indicated in Figure 2.8c - straight road and Figure 2.8d - curved road. The signs should be placed at the border between two tiles and should face towards oncoming traffic as indicated.

In these figures the arrow is the direction of the sign.

Street Name Signs

✎🔗

Signage

✎🔗

Liam

The raw materials as described in Unit D-1 - Duckietown parts

A set of signs to be used for assembling your Duckietown.

Making New Signage

✎🔗

If you find that what is available in the database in insufficient for your needs, then you will need to add to the existing database.

To do so you will have to load the original AprilTags file available here: pdf ps

Which tag you should use depends on what type of sign you are trying add. The ranges of tags are specified in Table 3.1.

April tag ID ranges
Purpose Size Family ID Range
Traffic signs 6.5cm x 6.5cm 36h11 1-199
Traffic lights 6.5cm x 6.5cm 36h11 200-299
Localization 6.5cm x 6.5cm 36h11 300-399
Street Name Signs 6.5cm x 6.5cm 36h11 400-587

First, find the last sign of the type that you are trying to make in the signs and tags doc. You will use the next available ID after this one.

Construct the new sign by first copying and pasting an existing sign of similar type, and then replacing/adding the new AprilTag. To add the new april tag, use a screen capture mathod to crop precisely around the tag at the top and sides and include the sign id at the bottom. Then paste the tag into your word file under your desired and resize it exactly 6.5cm (2.56inches).

If you make a new road name sign, you may need to change the font size of the name so that it appears on one line (this is why we like people with names like “ROY” and “RUS”).

Important: You must also add your new sign to the April Tags DB in the software repo.

Add a new block like the ones that already exists or modify the one with the appropriate tag id:

- tag_id: NEW_TAG_ID
  tag_type: in {TrafficSign, Light, Localization, StreetName}
  street_name: either NEW_STREET_NAME or blank
  vehicle_name: currently not used
  traffic_sign_type: either TRAFFIC_SIGN_TYPE or blank

The value of NEW_STREET_NAME is up to you to decide (have fun with it!). The value of TRAFFIC_SIGN_TYPE should be one of the signs in Figure 2.7

When finished, regenerate the PDF version of the Word file, and commit everything to the repo (via a pull request of course).

It is also possible of course to start you own completely different signs and tags database, but make sure that you specify in the april_tags code which database to load from.

Update the way that the april tags code loads the database

Duckietown Assembly

✎🔗

Shiying

Follow the rules in the Unit D-2 - Duckietown Appearance Specification.

comment

do we need this page at all?

Because of mathjax bug

Traffic lights Assembly

✎🔗

Marco Erni

Because of mathjax bug

Semantics of LEDS

✎🔗

???

headlights: white, constant

Assumption:

  • 20 fps to do LED detection

  • 1s to decide

  • 3 frequencies to detect

tail lights: red, 6 hz square wave

traffic light “GO” = green, 1 hz square wave

traffic light “STOP” = red, 1.5 Hz square wave

duckie light on top, state 0 = off

duckie light on top, state 1 = blue, 3 Hz, square wave

duckie light on top, state 2 = ?, 2.5 Hz square wave

duckie light on top, state 3 = ?, 2 Hz square wave

Because of mathjax bug

Duckiebot - DB17-lc configurations

✎🔗

This section contains the acquisition, assembly and setup instructions for the DB17-lc configurations. These instructions are separate from the rest as they approximately match the b releases of the Fall 2017 Duckietown Engineering Co. branches.

To understand how configurations and releases are defined, refer to: Unit C-1 - Duckiebot configurations.

Because of mathjax bug

Acquiring the parts (DB17-lc)

✎🔗

Upgrading your DB17 (DB17-wjd) configuration to DB17-lc (DB17-wjdlc) starts here, with purchasing the necessary components. We provide a link to all bits and pieces that are needed to build a DB17-lc Duckiebot, along with their price tag. If you are wondering what is the difference between different Duckiebot configurations, read Unit C-1 - Duckiebot configurations.

In general, keep in mind that:

  • The links might expire, or the prices might vary.
  • Shipping times and fees vary, and are not included in the prices shown below.
  • Buying the parts for more than one Duckiebot makes each one cheaper than buying only one.
  • A few components in this configuration are custom designed, and might be trickier to obtain.

A Duckiebot in DB17-wjd configuration.

Cost: USD 77 + Bumpers manufacturing solution

Time: 21 Days (LED board manufacturing and shipping time)

A kit of parts ready to be assembled in a DB17-lc configuration Duckiebot.

After receiving these components, you are ready to do some soldering before assembling your DB17-lc Duckiebot.

Soldering boards (DB17-l)

✎🔗

General rule in soldering

  • soldering the components according to the height of components - from lowest to highest

Shiying

Duckiebot DB17-l parts. The acquisition process is explained in Unit E-1 - Acquiring the parts (DB17-lc). The configurations are described in Unit C-1 - Duckiebot configurations.

Time: 30 minutes

A DB17-l Duckiebot

Assembling the Duckiebot (DB17-lc)

✎🔗

Shiying

Duckiebot DB17-lc parts. The acquisition process is explained in Unit E-1 - Acquiring the parts (DB17-lc).

Soldering DB17-lc parts. The soldering process is explained in Unit E-2 - Soldering boards (DB17-l).

Having assembled the Duckiebot in configuration DB17 (or any DB17-wjd). The assembly process is explained in Unit C-5 - Assembling the Duckiebot (DB17-jwd).

Time: about 30 minutes.

An assembled Duckiebot in configuration DB17-wjdlc.

Bumper Assembly

✎🔗

Duckiebot DB17-lc parts.

Having the Duckiebot with configuration DB17-wjd assembled. The assembly process is explained in Unit E-3 - Assembling the Duckiebot (DB17-lc).

Time: about 15 minutes.

A Duckiebot with Bumpers (configuration DB17-l2)

DB17-l setup

✎🔗

Shiying

Duckiebot DB17-lc parts. The acquisition process is explained in Unit E-1 - Acquiring the parts (DB17-lc).

Soldering DB17-lc parts. The soldering process is explained in Unit E-2 - Soldering boards (DB17-l).

Having assembled PWM Hat on the Duckiebot with configuration DB17-wjd. The assembly process is explained in Unit E-3 - Assembling the Duckiebot (DB17-lc).

Time: about 15 minutes.

A Duckiebot with LEDs attached (configuration DB17-l3)

Operation Manual - demos

✎🔗

This part describes a set of demos that demonstrate the functionality of the robots.

Some of these demos require different hardware on the robot, or in Duckietown.

Because of mathjax bug

Demo template

✎🔗

This is the template for the description of a demo.

First, we describe what is needed, including:

  • Robot hardware
  • Number of Duckiebots
  • Robot setup steps
  • Duckietown hardware

Duckiebot in configuration ???

Camera calibration completed.

Video of expected results

✎🔗

First, we show a video of the expected behavior (if the demo is succesful).

Duckietown setup notes

✎🔗

Here, describe the assumptions about the Duckietown, including:

  • Layout (tiles types)
  • Instrastructure (traffic lights, wifi networks, …) required
  • Weather (lights, …)

Do not write instructions here. The instructions should be somewhere in the part about Duckietowns. Here, merely point to them.

Duckiebot setup notes

✎🔗

Write here any special setup for the Duckiebot, if needed.

Do not write instructions here. The instructions should be somewhere in the appropriate setup part.

Pre-flight checklist

✎🔗

The pre-flight checklist describes the steps that are sufficient to ensure that the demo will be correct:

Check: operation 1 done

Check: operation 2 done

Demo instructions

✎🔗

Here, give step by step instructions to reproduce the demo.

Step 1: XXX

Step 2: XXX

Troubleshooting

✎🔗

Add here any troubleshooting / tips and tricks required.

Demo failure demonstration

✎🔗

Finally, put here a video of how the demo can fail, when the assumptions are not respected.

Because of mathjax bug

Lane following

✎🔗

This is the description of lane following demo.

Wheels calibration completed wheel calibration

Camera calibration completed Camera calibration

Joystick demo has been successfully launched Joystick demo

Duckiebot in configuration DB17-jwd

Calibrating the gain parameter to 0.6.

Video of expected results

✎🔗

Video of demo lane following

Duckietown setup notes

✎🔗

Assumption about Duckietown:

  • A duckietown with white and yellow lanes. No obstacles on the lane.
  • Layout conform to Duckietown Appearance Specifications
  • Required tiles types: straight tile, turn tile
  • Additional tile types:3-way/4-way intersection
  • Configurated Wifi network or Duckietown Wifi network

Environment of demo:

  • Good lightning

Duckiebot setup notes

✎🔗
  • Make sure the camera is heading ahead.
  • Duckiebot in configuration DB17-jwd
  • Calibrating the gain parameter to 0.6.

Pre-flight checklist

✎🔗

Check: Turn on joystick. Check: Turn on battery of the duckiebot. Check: Duckiebot drives correctly with joystick.

Demo instructions

✎🔗

Step 1: On duckiebot, in /DUCKIERTOWN_ROOT/ directory, run command:

duckiebot $ make demo-lane-following

Wait a while so that everything has been launched. Drive around with joystick to check if connection is working.

Step 2: Press X to recalibrate Anti-Instagram.

Step 3: Start the autonomous lane following by pressing the R1 button on joystick. to start autonomous.

Step 4: The Duckiebot should drive autonomously in the lane. Intersections and red lines are neglected and the Duckiebot will drive across them like it is a normal lane. Enjoy the demo.

Step 5: Stop the autonomous driving by pressing L1 button on the joystick and switch to joystick control.

Troubleshooting

✎🔗

Demo failure demonstration

✎🔗

If Anti-Instagram is badly calibrated, the duckiebot will not see enough line segments. This is especially a problem in the curve and the duckiebot will leave the lane. An example of this failure can be seen in this video for which we had a bad Anti-Instagram calibration. Hence, the Duckiebot sees not enough line segments and the lane following fails in the curve. To solve the problem Anti-Instagram needs to be relaunched. In the last part of the video the X button on the joystick is pressed and the Anti-Instagram node gets relaunched. We can see in RVIZ that the number of detected line segments gets increased drastically after the recalibration.

Because of mathjax bug

Demo Saviors

✎🔗

This is the description of the saviors obstacle avoidance demo.

Duckiebot in configuration DB17-wjd.

Camera calibration completed.

Wheel calibration completed.

Joystick demo successfully launched.

Video of expected results

✎🔗

Vimeo

As it can be inferred from the video, the duckiebot should stop if an obstacle (duckie or cone) is placed in front of the Duckiebot.

Duckietown setup notes

✎🔗

Duckietown built to specifications. No special requirements like april-tags, traffic lights or similar needed.

To demonstrate functionality, place obstacles (duckies S/M/L or cones) on driving lane. Best performance is achieved when obstacles are placed on the straights, not immediately after a curve.

Duckiebot setup notes

✎🔗

Currently the bot has to be on the devel-saviors-23feb branch on git. Furthermore the additional package sk-image has to be installed: 

duckiebot $ sudo apt-get install python-skimage

Pre-flight checklist

✎🔗

Check: Joystick is turned on

Check: Sufficient battery charge on duckiebot

Demo instructions

✎🔗

Step by step instructions to run demo:

Step 1: On the duckiebot, navigate to DUCKIETOWN_ROOT and run 

duckiebot $ source environment.sh

duckiebot $ catkin_make -C catkin_ws/

duckiebot $ make demo-lane-following

Wait for a couple of seconds until everything has been properly launched.

Step 2: In a second terminal on the duckiebot, run: 

duckiebot $ roslaunch obst_avoid obst_avoid_lane_follow_light.launch veh:=robot_name

This launches the obstacle avoidance node, wait again until it’s properly started up.

Step 3: Press the X button on the joystick to generate an anti-instagram transformation.

Within about the next 10 seconds in the terminal of Step2 this YELLOW message should appear:

!!!!!!!!!!!!!!!!!!!!TRAFO WAS COMPUTED SO WE ARE READY TO GO!!!!!!!!!!!!

Step 4: To (optionally) visualise output of the nodes run the following commands on your notebook:

laptop $ source set_ros_master.sh robot_name

laptop $ roslaunch obst_avoid obst_avoid_visual.launch veh:=robot_name

laptop $ rviz

Topics of interest are:

/robot_name/obst_detect_visual/visualize_obstacles (Markers which show obstacles, visualize via rviz!),

/robot_name/obst_detect_visual/image/compressed (Image with obstacle detection overlay, visualize via rqt!),

/robot_name/obst_detect_visual/bb_linelist (bounding box of obstacle detection, visualize via rqt),

/robot_name/duckiebot_visualizer/segment_list_markers (line segments).

Step 5: To drive press R1 to start lane following. Duckiebot stops if obstacle detected and in reach of the duckiebot. Removal of the obstacle should lead to continuation of lane following.

Troubleshooting

✎🔗

P: Objects aren’t properly detected, random stops on track.

S: Make sure that anti instagram was run properly. Repeat Step 3 if needed.

P: Duckiebot crashes obstacles.

S: Might be due to processes not running fast enough. Check if CPU load is too high, reduce if needed.

Further Reading

✎🔗

More information and details about our software packages can be found in our README on Github or in our Final Report.

Because of mathjax bug

Intersection Navigation

✎🔗

???

Because of mathjax bug

Indefinite Navigation

✎🔗

This is the description of the indefinite navigation demo.

Wheels calibration completed.wheel calibration

Camera calibration completed.Camera calibration

Joystick demo has been successfully launched.Joystick demo

Fully set up Duckietown (including April tags for intersections)

A motor gain of approximately 0.65 (strong influence in open-loop intersections)

Video of expected results

✎🔗

The video is at https://vimeo.com/247596730.

Demo: indefinite navigation

add a different video with an up to specification Duckietown.

Duckietown setup notes

✎🔗

A Duckietown with white and yellow lanes. No obstacles on the lane. Red stop lines at intersections. If several Duckiebots are present while running this demo, LEDs need to be installed for explicit communication and coordination of Duckiebots at intersections.

Duckiebot setup notes

✎🔗

Make sure the camera is securely tightened and properly calibrated.

Pre-flight checklist

✎🔗

Check: Joystick is turned on.

Check: Sufficient battery charge of the Duckiebot.

Check: Gain is set to approximately 0.65.

Demo instructions

✎🔗

Follow these steps to run the indefinite navigation demo on your Duckiebot:

Step 1: On the Duckiebot, navigate to the /DUCKIETOWN_ROOT/ directory, run the command:

duckiebot $ make indefinite-navigation

Wait until everything has been launched. Press X to start anti-instagram. Pressing R1 will start the autonomous lane following and L1 can be used to revert back to manual control.

In the current open-loop intersection navigation, no Duckiebot will successfully run the demo the first time. Parameter tuning is a must. The only two parameters that should be adjusted are the gain and trim, previously defined during the wheel calibration procedure.

The parameter pair which makes your bot go straight will unlikely work for the lane following due to the current controller design. Start with your parameter pair obtained from wheel calibration. If your Duckiebot stays too long on a curve during crossing an intersection, decrease your gain in steps of 0.05. If the Duckiebot doesn’t make the turn enough long, increase your gain in steps of 0.05.

Command to modify your gain (in this example to 0.65):

$ rosservice call /robot name/inverse_kinematics_node/set_gain -- 0.65
question

on laptop or bot?

Everything below is helpful for debugging if your robot does not follow the lane at all.

Step 2: Navigate to the Duckietown folder:

laptop $ cd ~/duckietown

then source the environment:

laptop $ source environment.sh

set the the ROS master to your vehicle:

laptop $ source set_ros_master.sh robot name

and finally launch rviz:

laptop $ rviz

In rviz, two markerarrays:

  • /robot name/duckiebot_visualizer/segment_list_markers, and
  • /robot name/lane_pose_visualizer_node/lane_pose_markers

can be visualized. The green arrow representing the pose estimate of the robot has to be in a reasonable direction.

Step 3: Always on the laptop, run:

laptop $ rqt

In rqt, the images can be visualized are:

  • /robot name/camera_node/image/compressed,
  • /robot name/line_detector_node/image_with_lines,
  • /robot name/lane_filter_node/belief_img.

Troubleshooting

✎🔗

Contact Julien Kindle(ETHZ) via Slack for further assistance.

Because of mathjax bug

Coordination

✎🔗

This demo (instructions from MIT2016) allow multiple Duckiebots that stop at an intersection to coordinate with LED signals and clear the intersection. Two demos are available, one for the case with traffic lights and one without them.

Duckiebot in configuration DB17-lc

Camera calibration and kinematic calibration completed.

Demo instructions

✎🔗

Troubleshooting

✎🔗
  • When it shows “Event:intersection_go”, the Duckiebot does not move. This problem is related to AprilTags.

Solution: go to config/baseline/pi_camera and change the framerate to 30 instead of 15.

  • [Warning] Topics ‘/![robot name]/camera_node/image/rect’ and ‘/![robot name]/camera_node/raw_camera_info’ do not appear to be synchronized. Solution:
Because of mathjax bug

Implicit Coordination

✎🔗

This demo allows multiple Duckiebots that stop at an intersection to coordinate themself without explicit communication and clear the intersection.

Duckiebot in configuration DB-wj

Camera calibration completed. camera calibration

Wheel calibration completed. wheel calibration

Object Detection setup.

Video of expected results

✎🔗

First, we show a video of the expected behavior (if the demo is succesful).

Duckietown setup notes

✎🔗

The Duckietown used for this demo has to be equipped with…

  • at least one intersection.
  • april tags providing information about the intersection type.

Duckiebot setup notes

✎🔗

No special setup is needed for the Duckiebot.

Laptop setup notes

✎🔗

Tensorflow needs to be installed:

Installing Tensorflow on Ubuntu

Inference model needs to be added:

Move to correct folder:

laptop $ cd ~/duckietown/catkin_ws/src/80-deep-learning/object_detection/models

Download model from here and place in the folder

Pre-flight checklist

✎🔗

Check: Joystick is turned on.

Check: Sufficient battery charge of the Duckiebot.

Demo instructions

✎🔗

Everything should be run from branch ‘devel-implicit-coordination-intersection’, don’t forget to build again.

Step 1: Place the participating Duckiebots on the ways to the same intersection with enough distance to recognize stop lines.

Step 2: Run the following commands on the Duckiebots:

Make sure you are in the Duckietown folder:

duckiebot $ cd ~/Duckietown

Run the demo:

duckiebot $ make demo-implicit_coordination

Step 3: Run the following commands on the laptop:

Make sure you are in the Duckietown folder:

laptop $ cd ~/Duckietown

Set up the environment:

laptop $ source environment.sh

Set your duckiebot as the ROS master:

laptop $ source set_ros_master.sh robot name

Run multivehicle detection:

laptop $ roslaunch duckietown multivehicle_detection.launch veh:=robot name

Step 4: Press R1 on your connected gamepad. The duckiebots now perform lane following until they stop at the intersection, where they coordinate themself and clear the intersection.

Troubleshooting

✎🔗

Demo failure demonstration

✎🔗

coming soon

Because of mathjax bug

Explicit Coordination

✎🔗

This demo allows different Duckiebots to coordinate at an intersection using LED-based communication. It handles three or four way intersections with or without a traffic light.

Duckiebot in configuration DB17-lc

Camera calibration and kinematic calibration completed.

Demo instructions

✎🔗

Troubleshooting

✎🔗
  • When it shows “Event:intersection_go”, the Duckiebot does not move. This problem is related to AprilTags.

Solution: go to config/baseline/pi_camera and change the framerate to 30 instead of 15.

  • [Warning] Topics ‘/![robot name]/camera_node/image/rect’ and ‘/![robot name]/camera_node/raw_camera_info’ do not appear to be synchronized. Solution:
Because of mathjax bug

Parallel Autonomy

✎🔗

???

Because of mathjax bug

Follow Leader

✎🔗

This is the description of the follow the leader demo.

Wheels calibration completed.wheel calibration

Camera calibration completed.Camera calibration

Lane following demo works.Lane following demo

Circle grid tags on Duckiebots.

More than one Duckiebot

Video of expected results

✎🔗

First, we show a video of the expected behavior (if the demo is successful).

Duckietown setup notes

✎🔗

A Duckietown with white and yellow lanes. No obstacles on the lane.

Duckiebot setup notes

✎🔗

Make sure the camera is heading ahead. Tighten the screws if necessary. Attach circle grid with tape at the rear. A pdf file with circle grids in the right size (0.0125m between dot centers) can be found in /DUCKIETOWN_ROOT/catkin_ws/src/50-misc-additional-functionality/vehicle_detection/CircleGrid.pdf.

Duckiebot with Circle Grid

Pre-flight checklist

✎🔗

Check: Joystick is turned on.

Check: Sufficient battery charge of the Duckiebot.

Demo instructions

✎🔗

Step 1: Pull the branch devel-implicit-coord-formationKeeping from the Duckietown repository and run catkin_make.

Place more than one Duckiebot on the outer lanes of Duckietown, such that the Duckiebots drive an circle around Duckietown with lane following. The distance between the Duckiebots should be more than 0.5m to start the demo successfully.

Run the following commands on different Duckiebots at the same time.

Step 1: On Duckiebot, in /DUCKIETOWN_ROOT/ directory, checkout the branch devel-implicit-coord-formationKeeping and run the following commands:

duckiebot $ catkin_make -C catkin_ws/

duckiebot $ make demo-vehicle_follow_leader

Wait a while so that everything has been launched. Press R2 to start autonomous lane following with vehicle avoidance. Press L1 to switch to joystick control or R1 to switch to lane following without vehicle avoidance. In autonomous lane following mode with vehicle avoidance the Duckiebot is keeping a distance to another Duckiebot on front of him.

Step 2: On laptop, make sure ros environment has been activated.

laptop $ rqt_image_view

In rqt_image_view the images with circle grid are visualized: /(vehicle_name)/vehicle_detection_node/circlepattern_image.

If the circle grid is detected, the grid is drawn in the image.

Troubleshooting

✎🔗

Add here any troubleshooting / tips and tricks required.

Demo failure demonstration

✎🔗

Finally, put here a video of how the demo can fail, when the assumptions are not respected.

Because of mathjax bug

Preliminaries

✎🔗

to write

Because of mathjax bug

Chapter template

✎🔗

Theory chapters benefit from a standardized exposition. Here, we define the template for these chapters. Rememeber to check Unit B-2 - Basic Markduck guide for a comprehensive and up-to-date list of Duckiebook supported features.

Example Title: PID control

✎🔗

Start with a brief introduction of the discussed topic, describing its place in the bigger picture, justifying the reading constraints/guidelines below. Write it as if the reader knew the relevant terminology. For example:

PID control is the simplest approach to making a system behave in a desired way rather than how it would naturally behave. It is simple because the measured output is directly feedbacked, as opposed to, e.g., the system’s states. The control signal is obtained as a weighted sum of the tracking error (_P_roportional term), its integral over time (_I_ntegrative term) and its instantaneous derivative (_D_erivative term), from which the appellative of PID control. The tracking error is defined as the instantaneous difference between a reference and a measured system output.

Knowledge necessary:

Required Reading: Insert here a list of topics and suggested resources related to necessary knowledge in order to understand the content presented. Example:

Terminology: autonomy overview

System Modeling: basic kinematics, basic dynamics, linear algebra, State space representations, Linear Time Invariant Systems

Suggested Reading: Insert here a list of topics and suggested resources related to recommended knowledge in order to better understand the content presented. Example:

Definitions of Stability, Performances and Robustness: [7], …

observability/detectability and controllability/reachability: [7]

Discrete time PID: [7]

Bode diagrams: [7]

Nyquist plots: [7]

[…]

Problem Definition

✎🔗

In this section we crisply define the problem object of this chapter. It serves as a very brief recap of exactly what is needed from previous atoms as well. E.g.

Let:

\begin{align} \dot{\state}_t = A\state_t+Bu_t \\ y = C\state_t+Du_t \label{eq:system}\tag{1} \end{align}

be the LTI model of the Duckiebot’s plant, with $x \in \statesp$, $y \in \mathbb{R}^p$ and $u \in \mathbb{R}^m$. We recall (Duckiebot Modeling) that:

\begin{align} A &= \left[ \begin{array}{ccc} a_{11} & \dots & a_{1n} \\ \vdots & \ddots & \vdots \\ a_{n1} & \dots & a_{nn} \end{array} \right] \\ B &= \left[ b_1 \,\, \dots \,\, b_m \right]^T \\ C &= \left[ c_1 \ \,\, \dots \,\, c_p \right] \\ D &= 0. \end{align}

[…]

Remember you can use the problem environment of $\LaTeX$ to formally state a problem:

PID Given a system \eqref{eq:system} and measurements of the output $\tilde{y}_t = y_t + n_t, n_t \sim \cal{N}(0,\sigma)$, find a set of PID coefficients that meet the specified requirements for: - stability, - performance, - robustness.

as shown in (Figure 1.1).

A classical block diagram for PID control. We like to use a lot of clear figures in the Duckiebook.

Introduced Notions

✎🔗

Examples

✎🔗

This section serves as a collection of theoretical and practical examples that can clarify part or all of the above.

Theoretical Examples

✎🔗

More academic examples

✎

Immagine a spring-mass-damper system…

✎

[…]

Implementation Examples

✎🔗

More Duckiebot related examples

✎ ✎

[…]

Pointers to Exercises

✎🔗

Here we just add references to the suggested exercises, defined in the appropriate exercise chapters.

References

✎🔗

Do not include a reference chapter. References are automatically compiled to the Bibliography Section.

Jacopo

Jacopo

Jacopo

Because of mathjax bug

Symbols and conventions

✎🔗

Andrea

Sets

✎🔗

Dzenan

k:sets

Maps

✎🔗

k:sets

k:maps

Numbers

✎🔗

k:sets

k:naturals, k:integers, k:reals

Natural numbers

✎🔗

$\nats = \{0, 1, 2, \cdots\}$

The natural numbers are the set positive numbers, including zero.

Given two natural their addition is always a natural number:

$$ a+b = c \in \nats, \forall a,b \in \nats. \label{eq:intro-nats}\tag{1}$$

The same does not hold of the subtraction operation:

$$ a-b = c \in \nats \iff a \geq b. $$

For this reason set of integer numbers is defined.

Complex numbers

✎🔗

Dzenan

Linearity and Vectors

✎🔗

Jacopo

Linear algebra provides the set of mathematical tools to (a) study linear relationships and (b) describe linear spaces. It is a field of mathematics with important ramifications.

Linearity is an important concept because it is powerful in describing the input-output behavior of many natural phenomena (or systems). As a matter of fact, all those systems that cannot be modeled as linear, still can be approximated as linear to gain an intuition, and sometimes much more, of what is going on.

So, in a way or the other, linear algebra is a starting point for investigating the world around us, and Duckietown is no exception.

This chapter is not intended to be a comprehensive compendium of linear algebra.

this reference

this other reference

add references throughout all chapter

Real numbers are complex for you?: Number theory addref

$\forall$ is a typo for A and $\in$ are Euros? Mathematical symbolic language.

Linearity

✎🔗

In this section we discuss vectors, matrices and linear spaces along with their properties.

Before introducing the these arguments, we need to formally define what we mean by linearity. The word linear comes from the latin linearis, which means pertaining to or resembling a line. You should recall that a line can be represented by an equation like $y = mx + q$, but here we intend linearity as a property of maps, so there is a little more to linearity than lines (although lines are linear maps indeed).

To avoid confusions, let us translate the concept of linearity in mathematical language.

Linearity A function $f: \aset{X} \to \aset{Y}$ is linear when, $\forall x_i \in \aset{X}$, $i = \{1,2\}$, and $\forall a \in \reals$:

\begin{align} f(ax_1) &= af(x_1), \label{eq:lin1}\tag{2} \quad \text{and:} \\ f(x_1 + x_2) &= f(x_1) + f(x_2) \label{eq:lin2}\tag{3} \end{align}

Condition \eqref{eq:lin1} is referred to as the property of homogeneity (of order 1), while condition \eqref{eq:lin2} is referred to as additivity.

Superposition Principle Conditions \eqref{eq:lin1} and \eqref{eq:lin2} can be merged to express the same meaning through: \begin{align} f(ax_1 + bx_2) = af(x_1) + bf(x_2), \forall x_i \in \aset{X}, i = \{1,2\}, \forall a,b \in \reals \label{eq:linearity}\tag{4}. \end{align}

This equivalent condition \eqref{eq:linearity} is instead referred to as superposition principle, which unveils the bottom line of the concept of linearity: adding up (equivalently, scaling up) inputs results in an added up (equivalently, scaled up) output.

Vectors

✎🔗

Let $n$ belong to the set of natural numbers $\nats$, i.e., $n \in \nats$, and let $a_i \in \reals$, $i = \{1, \dots, n\}$ be real coefficients. While $\reals$ is the set of real numbers, $\reals^n$ is the set of all $n$-tuples of real numbers.

Vector and components

An $n$-dimensional $\textit{vector}$ is an $n$-tuple:

\begin{align} \label{eq:vector}\tag{5} \avec{v} = \left[ \begin{array}{c} v_1 \\ \vdots \\ v_n \end{array} \right] \in \reals^{n \times 1} \equiv \reals^n, \end{align}

of components $v_1, \dots, v_n \in \reals$.

Vector notation A more general notation for tuples can be used when denoting vectors: \begin{align} \label{eq:vector-tuple-notation}\tag{6} \avec{v} = \tup{v_1, \cdots, v_n}. \end{align} In these preliminaries, we will adopt the \eqref{eq:vector} “engineering” notation as it arguably simplifies remembering vector-matrix operations (Unit G-9 - Matrices and vectors).

You can imagine a vector Figure 6.1 as a “directional number”, or an arrow that starts a certain point and goes in a certain direction (in $\reals^n$). In this representation, the number is the length of the arrow, or the magnitude of the vector (sometimes referred to even as modulus), and it can be derived through the vector’s components.

Length of a vector We define the length, or modulus, of a vector $\avec{v} \in \reals^n$ as: \begin{align} \label{eq:vec-2-norm}\tag{7} \|\avec{v}\| = \sqrt{v_1^2 + \dots + v_n^2} \in \reals. \end{align}

2-norm Generally speaking, it is not always possible to define the length of a vector (addref). But when it is possible (e.g., in Hilbert spaces), and in Duckietown it always is, there are many ways to define it. The most common and intuitive definition is the Euclidian- or 2-norm, which is defined above in \eqref{eq:vec-2-norm}.

We will discuss norms more in detail in Unit G-13 - Norms.

Unit vector A unit vector, or versor, is a vector $\avec{e}$ of of unit length: \begin{align} \label{eq:unit-vector}\tag{8} \|\avec{e}\| = 1. \end{align}

Unit vectors are used to define the directions of the components of a vector, allowing for an algebraic rather than vectorial representation. As we will see in Subsection 6.2.1 - Vector algebra, this will make the algebra of vectors more intuitive.

A vector, its components expressed as multiples of unit vectors.

Let $\avec{v} \in \reals^3$ be a vector defined in the Cartesian space. Let, moreover, $(\avec{i},\avec{j},\avec{k})^T$ be the versor of the Cartesian axis, i.e.: \begin{align}\label{eq:example-vector-algebraic}\tag{9} \avec{i} &= [1,0,0]^T; \\ \avec{j} &= [0,1,0]^T; \\ \avec{k} &= [0,0,1]^T. \end{align} Then, a vector can be written equivalently in vector or algebraic form: $\avec{v} = [v_1, v_2, v_3]^T = v_1\avec{i} + v_2\avec{j}+v_3\avec{k}$. Unit vectors are sometimes explicitly denoted with a hat (^), e.g., $\hat{\avec{i}}, \hat{\avec{j}}, \hat{\avec{k}}$.

Normalizing vectors Every vector can be made into a unit vector, or normalized, by dividing each of its components by the vector’s magnitude: \begin{align}\label{eq:vector-normalizing}\tag{10} \hat{\avec{v}} = \frac{\avec{v}}{\|\avec{v}\|} = \left[\frac{v_1}{\|\avec{v}\|}, \frac{v_2}{\|\avec{v}\|}, \frac{v_3}{\|\avec{v}\|}\right]^T. \end{align}

Vector algebra

✎🔗

We here define operations amongst two given vectors defined in the same space: $\avec{u} = [u_1, u_2, u_3]^T, \avec{v} = [v_1, v_2, v_3]^T \in \reals^3$.

Vectorial Sum

✎🔗

The sum of two vectors is a vector, and its components are the sum of the two vectors components.

Vectorial sum \begin{align} \label{eq:vector-sum}\tag{11} \avec{u} + \avec{v} = [u_1+v_1, u_2+v_2, u_3+v_3]^T. \end{align}

Sum Mathematical operations come in pairs, which represent the same concept. A sum operation, sometimes more extensively referred to as the algebric sum, is the concept of summing, i.e., it includes both addition and subtraction. (A subtraction is nothing but an addition between positive and negative numbers.)

The parallelogram law helps visualize the results of the vectorial sum operation Figure 6.2.

The sum of two vectors can be visualized with the parallelogram law.

Dot, or scalar, product

✎🔗

The dot, or scalar, product of two vectors ($\avec{u}$,$\avec{v} \in \mathbb{R}^3$) is a scalar ($a \in \mathbb{R}$) equal to the sum of the products of the components of the vectors. Equivalently, it can be expressed as the product of the magnitudes of the two vectors times the cosine of the angle between them, $\phi \in [0,2\pi)$.

Scalar product \begin{align} \label{eq:vector-dot-product}\tag{12} \avec{u} \cdot \avec{v} = u_1v_1+u_2v_2+u_3v_3 = \|u\|\|v\|\cos(\phi) \in \mathbb{R} \end{align}

The dot product is a measure of the projection of vectors on one another (Figure 6.3).

When the two vectors are perpendicular, or orthogonal, the dot product is zero ($\cos(\pi/2) = 0$). This fact is often used as a test for orthogonality. Orthogonality is an important concept for linear spaces, as the most “efficient” basis are orthogonal.

The scalar product between two vectors measures the projection of one on each other.

Cross, or vector, product

✎🔗

While the dot product depends on the metric chosen in the space (the Euclidian norm, in our case), the cross product even requires the definition of an orientation, or handedness.

Standard Basis In the Euclidian space $\mathbb{R}^3$, $\hat{\avec{i}}, \hat{\avec{j}}, \hat{\avec{k}}$ are the unit vectors for the standard basis, which is right handed.

In a right handed reference system such as the standard basis, the right hand rule (Figure 6.4) is the handy-est way to identify the direction of the vector resulting from a cross product.

ProTip: There is a valid reason for which it is called the right hand rule. Don’t use your left hand because you are holding a pen with the right one.

The right hand rule points in the direction of the resulting vector from a cross product.

The cross, or vector, product between two vectors ($\avec{u}$, $\avec{v} \in \mathbb{R}^3$) is a vector that is orthogonal to each of the two vectors, hence is normal, or perpendicular, to the plane containing them. Its magnitude is given by the product of their magnitude times the sine of the angle between them, and its direction is indicated by the normal unit vector ($\hat{\avec{n}} \in \mathbb{R}^3$), identified by the right hand rule.

Vector product \begin{align} \label{eq:vector-cross-product}\tag{13} \avec{u} \times \avec{v} = [u_2v_3-u_3v_2, u_3v_1-u_1v_3, u_1v_2-u_2v_1]^T = \|u\|\|v\|\sin(\phi) \hat{\avec{n}}. \end{align}

Geometric interpretation A cross product encodes two pieces on information: a direction, which is orthogonal to the plane spanned by the two vectors, and a magnitude, which is equal to the area of the parallelogram having $\avec{u}$, and $\avec{v}$ as sides.

Keeping \eqref{eq:vector-cross-product} and Remark 7 - Geometric interpretation in mind, it should be intuitive to understand that: \begin{align} \label{eq:vec-cross-vv-v0}\tag{14} \avec{v} \times \avec{v} &= \avec{0}, \forall \avec{v} \in \mathbb{R}^n, \\ \avec{v} \times \avec{0} &= \avec{0}, \forall \avec{v} \in \mathbb{R}^n. \end{align}

The zero vector ($\avec{0}$) is a vector with zero magnitude, not the same as the number zero ($0$).

Each component of $\avec{w}$ is the difference of the products of the two other components of $\avec{u}$, and $\avec{v}$, in the order given by the chosen handedness of the basis. This combination resembles a cross (Figure 6.5), from which the name of cross product.

Each component of the resulting vector is the product of the alternated other components, forming a cross.

The components of a cross product can be computed through the Sarrus rule (see Section 7.6 - Determinant).

As consequence of the vectorial product’s definition and right handedness of the basis, the following hold true in the Cartesian space:

\begin{align} \label{eq:sb-cross-products}\tag{15} \hat{\avec{i}} \times \hat{\avec{j}} &= \hat{\avec{k}} \\ \hat{\avec{j}} \times \hat{\avec{k}} &= \hat{\avec{i}} \\ \hat{\avec{k}} \times \hat{\avec{i}} &= \hat{\avec{j}}. \end{align}

Properties of vectors

✎🔗

In this section we highlight the properties of vector operations, that derive from their definitions.

✎

The vector sum obejs the following:

  • $\avec{u} + \avec{v} = \avec{v} + \avec{u}$,
  • $(\avec{u} + \avec{v}) + \avec{w} = \avec{u} + (\avec{v} + \avec{w})$,
  • $a(\avec{u} + \avec{v}) = a\avec{u} + a\avec{v}$,
  • $(a+b)\avec{u} = a\avec{u} + b\avec{u}$,
  • $\avec{u} + \avec{0} = \avec{u}$, therefore $\avec{u} + (-\avec{u}) = \avec{0}$.
✎

Letting $\phi \in [0,2\pi)$ be the angle between two vectors $\avec{u}, \avec{v}$, the dot product obejs the following:

  • $\avec{u} \cdot \avec{v} = \| \avec{u} \|\| \avec{v} \|\cos(\phi)$,
  • $\avec{u} \cdot \avec{u} = \| \avec{u} \|^2$,
  • $\avec{u} \cdot \avec{v} = \avec{v} \cdot \avec{u}$,
  • $\avec{u} \cdot (\avec{v} + \avec{w}) = \avec{u} \cdot \avec{v} + \avec{u} \cdot \avec{w}$,
  • $a (\avec{u} \cdot \avec{v}) = (a\avec{u}) \cdot \avec{v}$,
  • $\avec{0} \cdot \avec{u} = 0$
  • $\avec{u} \cdot \avec{v}$ = 0 $\iff$ $\avec{u}=\avec{0}$, $\avec{v}=\avec{0}$, or $\avec{u} \bot \avec{v}$.
✎

Letting $\phi \in [0,2\pi)$ be the angle between two vectors $\avec{u}, \avec{v}$, the cross product obejs the following:

  • $\avec{u} \times \avec{v} = \| \avec{u} \|\| \avec{v} \|\sin(\phi) \hat{\avec{n}}$,
  • $\avec{u} \times \avec{v} = - \avec{v} \times \avec{u}$,
  • $(a\avec{u}) \times \avec{v} = \avec{u} \times (a\avec{v}) = a(\avec{u} \times \avec{v})$,
  • $\avec{u} \times (\avec{v} + \avec{w}) = \avec{u} \times \avec{v} + \avec{u} \times \avec{w}$,
  • $\avec{u} \cdot (\avec{v} \times \avec{w}) = (\avec{u} \times \avec{v}) \cdot \avec{w}$,
  • $\avec{u} \times (\avec{v} + \avec{w}) = (\avec{w} \cdot \avec{u}) \avec{v} - (\avec{v} \cdot \avec{u}) \avec{w} \neq (\avec{u} \times \avec{v}) + \avec{w}$,
  • $\avec{u} \times \avec{v} = 0 \iff \avec{u}=\avec{0}, \avec{v}=\avec{0}$, or $\avec{u} \parallel \avec{v}$.

Linear dependance

✎🔗

Linear dependance Two or more vectors $\{\avec{v_1},\cdots,\avec{v_n}\}$ are linearly dependant if there exists a set of scalars $\{a_1, \cdots, a_k\}, k \leq n$, that are not all zero, such that: $$ \label{eq:lin-dep}\tag{1} a_1\avec{v_1} + \cdots + a_k\avec{v_k} = \avec{0}. $$

When \eqref{eq:lin-dep} is true, it is possible to write at least one vector as a linear combination of the others.

Linear independance Two or more vectors ${\avec{v_1},\cdots,\avec{v_n}}$ are linearly independant if \eqref{eq:lin-dep} can be satisfied only by $k=n$ and $a_i =0, \forall i = 1, \cdots, n$.

Pointers to Exercises

✎🔗

Here we just add references to the suggested exercises, defined in the appropriate exercise chapters.

add exercises

Matrices basics

✎🔗

Dzenan

k:basic_math

k:linear_algebra

k:matrices

A matrix:

$$ \amat{A} = \left[ \begin{array}{ccc} a_{11} & \dots & a_{1n} \\ \vdots & \ddots & \vdots \\ a_{m1} & \dots & a_{mn} \end{array} \right] \in \reals^{m \times n} \label{eq:matrix}\tag{1} $$

is a table ordered by ($m$) horizontal rows and ($n$) vertical columns. Its elements are typically denoted with lower case latin letters, with subscripts indicating their row and column respectively. For example, $a_{ij}$ is the element of $A$ at the $i$-th row and $j$-th column.

A matrix. This image is taken from [17]

A vector is a matrix with one column.

Matrix dimensions

✎🔗

The number of rows and columns of a matrix are referred to as the matrix dimensions. $\amat{A} \in \reals^{m \times n}$ has dimensions $m$ and $n$.

Fat matrix When $n \gt{} m$, i.e., the matrix has more columns than rows, $\amat{A}$ is called fat matrix.

Tall matrix When $n \lt{} m$, i.e., the matrix has more rows than columns, $\amat{A}$ is called tall matrix.

Fat matrix When $n = m$, $\amat{A}$ is called square matrix.

Square matrices are particularly important.

Diagonal matrix

✎🔗

Diagonal matrix A diagonal matrix has non zero elements only on its main diagonal. \begin{align} \amat{A} = \left[ \begin{array}{ccc} a_11 & 0 & \dots & 0 \\ 0 & a_22 & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 \\ 0 & \dots & 0 & a_nn \end{array} \right] \end{align}

Identity matrix

✎🔗

Identity matrix An identity matrix is a diagonal square matrix with all elements equal to one. \begin{align} \amat{I} = \left[ \begin{array}{ccc} 1 & 0 & \dots & 0 \\ 0 & 1 & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 \\ 0 & \dots & 0 & 1 \end{array} \right] \end{align}

Null matrix

✎🔗

Null matrix The null, or Zero, matrix is a matrix whos elements are all zeros. \begin{align} \amat{0} = \left[ \begin{array}{ccc} 0 & \dots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \dots & 0 \end{array} \right] \end{align}

Determinant

✎🔗
  • 2x2
  • 3x3
  • nxn

Rank of a matrix

✎🔗

Trace of a matrix

✎🔗
Because of mathjax bug

Matrix inversions

✎🔗

Dzenan

Matrix Inverse

✎🔗

Square matrix:

$\amat{A}\amat{A}^{-1} = \amat{I}$

Matrices and vectors

✎🔗

Dzenan

  • matrix-vector product

Matrix as representation of linear (vector) spaces

✎🔗
  • linear system to matrix representation
  • linearly dependent and independent spaces

Fundamental spaces

✎🔗
  • Null space
  • Range/image

Eigenvalues and Eigenvectors

✎🔗
  • for square matrices

  • for rectangular matrices (topic for advanced-linear-algebra?)

  • condition number of a matrix (?)

Because of mathjax bug

Matrix operations (basic)

✎🔗

Dzenan

  • sum of matrices
  • product of matrices
  • matrix transpose – Symmetric matrix {#mat-sym}
  • matrix concatenation

Properties

✎🔗
Because of mathjax bug

Matrix operations (complex)

✎🔗

Dzenan

  • matrix scalar product
  • matrix Hadamart product
  • matrix power
  • matrix exponential

Properties

✎🔗
Because of mathjax bug

Matrix diagonalization

✎🔗

Dzenan

  • singular value decomposition SVD (topic for advanced-linear-algebra?)
✎
  • show how to diagonalize matrices and why it is relevant (it will come in handy for state space representation chapter chapter)

Norms

✎🔗

Dzenan

finish writing

Other metrics can be defined to measure the “length” of a vector. Here, we report some commonly used norms. For a more in depth discussion of what constitutes a norm, and their properties, see:

[18].

$p$-norm

✎🔗

Let $p \geq 1 \in \reals$. The $p$-norm is defined as:

$p$-norm \begin{align} \label{eq:vec-p-norm}\tag{1} \|\avec{v}\|_p = \displaystyle \left( \sum_{i=1}^{n} |v_i|^p \right)^{\frac{1}{p}}. \end{align}

The $p$-norm is a generalization of the $2$-norm ($p=2$ in \eqref{eq:p-norm}) introduced above (Definition 19 - Length of a vector). The following $1$-norm and $\infty$-norm can as well be obtained from \eqref{eq:vec-p-norm} with $p=1$ and $p \rightarrow \infty$ respectively.

One norm

✎🔗

The $1$-norm is the sum of the absolute values of a vector’s components. It is sometimes referred to as the Taxicab norm, or Manhattan distance as it well describes the distance a cab has to travel to get from a zero starting point to a final destination $v_i$ on a grid.

$1$-norm Given a vector $\avec{v} \in \reals^n$, the $1$-norm is defined as: \begin{align} \label{eq:vec-one-norm}\tag{2} \|\avec{v}\| = \displaystyle \sum_{i=1}^{n}|v_i|. \end{align}

$\infty$-norm

✎🔗

The infinity norm measures the maximum component, in absolute value, of a vector.

$\infty$-norm \begin{align} \label{eq:vec-inf-norm}\tag{3} \|\avec{v}\| = \displaystyle \max(|v_1|, \cdots, |v_n|). \end{align}

Definition

✎🔗

Properties

✎🔗
Because of mathjax bug

From ODEs to LTI systems

✎🔗

Miguel

Linearization

✎🔗

Miguel

Probability basics

✎🔗

In this chapter we give a brief review of some basic probabilistic concepts. For a more in-depth treatment of the subject we refer the interested reader to a textbook such as [19].

Random Variables

✎🔗

The key underlying concept in probabilistic theory is that of an event, which is the output of a random trial. Examples of an event include the result of a coin flip turning up HEADS or the result of rolling a die turning up the number “4”.

Random Variable A (either discrete or continuous) variable that can take on any value that corresponds to the feasible output of a random trial.

For example, we could model the event of flipping a fair coin with the random variable $X$. We write the probability that $X$ takes HEADS as $p(X=\text{HEADS})$. The set of all possible values for the variable $X$ is its domain, $\aset{X}$. In this case, $$ \aset{X}=\{\text{HEADS},\text{TAILS}\}. $$ Since $X$ can only take one of two values, it is a binary random variable. In the case of a die roll, $$ \aset{X}=\{1,2,3,4,5,6\}, $$ and we refer to this as a discrete random variable. If the output is real value or a subset of the real numbers, e.g., $\aset{X} = \reals$, then we refer to $X$ as a continuous random variable.

Consider once again the coin tossing event. If the coin is fair, the have $p(X=HEADS)=p(X=TAILS)=0.5$. Here, the function $p(x)$ is called the probability mass function or pmf. The pmf is shown in Figure 16.1.

The pmf for a fair coin toss

Here are some very important properties of $p(x)$: - $0\leq p(x) \leq (1)$ - $\sum_{x\in\aset{X}}=1$

In the case of a continuous random variable, we will call this function $f(x)$ and call it a probability density function, or pdf.

In the case of continuous RVs, technically the $p(X=x)$ for any value $x$ is zero since $\aset{X}$ is infinite. To deal with this, we also define another important function, the cumulative density function, which is given by $F(x) \triangleq p(X\leq x)$, and now we can define $f(x) \triangleq \frac{d}{dx}F(x)$. A pdf and corresponding cdf are shown in Figure 16.2 (This happens to be a Gaussian distribution, defined more precisely in Subsection 16.1.8 - The Gaussian Distribution.).

The continuous pdf and cdf

Joint Probabilities

✎🔗

If we have two different RVs representing two different events $X$ and $Y$, then we represent the probability of two distinct events $x \in \aset{X}$ and $y \in \mathcal{Y}$ both happening, which we will denote as following: $p(X=x \; \text{AND} \; Y=y) = p(x,y)$. The function $p(x,y)$ is called joint distribution.

Conditional Probabilities

✎🔗

Again, considering that we have to RVs, $X$ and $Y$, imagine these two events are linked in some way. For example, $X$ is the numerical output of a die roll and $Y$ is the binary even-odd output of the same die roll. Clearly these two events are linked since they are both uniquely determined by the same underlying event (the rolling of the die). In this case, we say that the RVs are dependent on one another. In the event that we know one of events, this gives us some information about the other. We denote this using the following conditional distribution $p(X=x \; \text{GIVEN} \; Y=y) \triangleq p(x|y)$.

Write down the conditional pmf for the scenario just described assuming an oracle tells you that the die roll is even. In other words, what is p(x|\text{EVEN})?

(Warning: if you think this is very easy that’s good, but don’t get over-confident.)

The joint and conditional distributions are related by the following (which could be considered a definition of the joint distribution):

\begin{equation} p(x,y) = p(x|y)p(y) \label{eq:joint}\tag{1} \end{equation}

and similarly, the following could be considered a definition of the conditional distribution:

\begin{equation} p(x|y) = \frac{p(x,y)}{p(y)} \; \text{if} \; p(y) \gt{} 0 \label{eq:condition}\tag{2} \end{equation}

In other words, the conditional and joint distributions are inextricably linked (you can’t really talk about one without the other).

If two variables are independent, then the following relation holds: $p(x,y)=p(x)p(y)$.

Bayes’ Rule

✎🔗

Upon closer inspection of \eqref{eq:joint}, we can see that the choice of which variable to condition upon is completely arbitrary. We can write:

$$ p(y|x)p(x) = p(x,y) = p(x|y)p(y) $$

and then after rearranging things we arrive at one of the most important formulas for mobile robotics, Bayes’ rule:

\begin{equation} p(x|y) = \frac{p(y|x)p(x)}{p(y)} \label{eq:bayes}\tag{3} \end{equation}

Exactly why this formula is so important will be covered in more detail in later sections (TODO), but we will give an initial intuition here.

Consider that the variable $X$ represents something that we are trying to estimate but cannot observe directly, and that the variable $Y$ represents a physical measurement that relates to $X$. We want to estimate the distribution over $X$ given the measurement $Y$, $p(x|y)$, which is called the posterior distribution. Bayes’ rule lets us to do this. For every possible state, you take the probability that this measurement could have been generated, $p(y|x)$, which is called the measurement likelihood, you multiply it by the probability of that state being the true state, $p(x)$, which is called the prior, and you normalize over the probability of obtaining that measurement from any state, $p(y)$, which is called the evidence.

From Wikipedia: Suppose a drug test has a 99% true positive rate and a 99% true negative rate, and that we know that exactly 0.5% of people are using the drug. Given that a person’s test gives a positive result, what is the probability that this person is actually a user of the drug.

Answer: $\approx$ 33.2%. This answer should surprise you. It highlights the power of the prior.

Conditional Independence

✎🔗

Two RVs, $X$ and $Y$ may be correlated, we may be able to encapsulate the dependence through a third random variable $Z$. Therefore, if we know $Z$

A graphical representation of the conditional independence of $X$ and $Y$ given $Z$
comment

Is there a discussion of graphical models anywhere? Doing a good job of sufficiently describing graphical models and the dependency relations that they express requires careful thought. Without it, we should refer readers to a graphical models text (e.g., Koller and Friedman, even if it is dense)

Moments

✎🔗

The $n$th moment of an RV, $X$, is given by $E[X^n]$ where $E[]$ is the expection operator with:

$$ E[f(X)] = \sum_{\aset{X}} x \, f(x) $$ in the discrete case and $$ E[f(X)] = \int x\, f(x) dx $$ in the continuous case.

The 1st moment is the mean, $\mu_X=E[X]$.

The $n$th central moment of an RV, $X$ is given by $E[(X-\mu_X)^n]$. The second central moment is called the covariance, $\sigma^2_X=E[(X-\mu_X)^2]$.

Entropy

✎🔗

The entropy of an RV is a scalar measure of the uncertainty about the value the RV.

A common measure of entropy is the Shannon entropy, whose value is given by

\begin{equation} H(X)=-E[\log_2 p(x)] \label{eq:shannon}\tag{4} \end{equation}

This measure originates from communication theory and literally represents how many bits are required to transmit a distribution through a communication channel. For many more details related to information theory we recommend [20].

As an example, we can easily write out the Shannon entropy associated with a binary RV (e.g. flipping a coin) as a function of the probability that the coin turns up heads (call this $p$):

\begin{equation} H(X) = -p\log_2 p - (1-p)\log_2 (1-p) \label{eq:binary_entropy}\tag{5} \end{equation}

The Shannon entropy of a binary RV $X$

Notice that our highest entropy (uncertainty) about the outcome of the coin flip is when it is a fair coin (equal probability of heads and tails). The entropy decays to 0 as we approach $p=0$ and $p=1$ since in these two cases we have no uncertainty about the outcome of the flip. It should also be clear why the function is symmetrical around the $p=0.5$ value.

The Gaussian Distribution

✎🔗

In mobile robotics we use the Gaussian, or normal, distribution a lot.

comment

The banana distribution is the official distribution in robotics! - AC

comment

The banana distribution is Gaussian! http://www.roboticsproceedings.org/rss08/p34.pdf - LP

The 1-D Gaussian distribution pdf is given by:

\begin{equation} \mathcal{N}(x|\mu,\sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}}e^{-\frac{1}{2\sigma^2}(x-\mu)^2} \label{eq:gaussian1D}\tag{6} \end{equation}

where $\mu$ is called the mean of the distribution, and $\sigma$ is called the standard deviation. A plot of the 1D Gaussian was previously shown in Figure 16.2.

We will rarely deal with the univariate case and much more often deal with the multi-variate Gaussian:

\begin{equation} \mathcal{N}(\state|\bmu,\bSigma) = \frac{1}{(2*\pi)^{D/2}|\bSigma|^{1/2}}\exp[-\frac{1}{2}(\state-\bmu)^T\bSigma^{-1}(\state - \bmu)] \label{eq:gaussianND}\tag{7} \end{equation}

The value from the exponent: $(\state-\bmu)^T\bSigma^{-1}(\state - \bmu)$ is sometimes written $||\state - \bmu||_\bSigma$ and is referred to as the Mahalanobis distance or energy norm.

Mathematically, the Gaussian distribution has some nice properties as we will see. But is this the only reason to use this as a distribution. In other words, is the assumption of Gaussianicity a good one?

There are two very good reasons to think that the Gaussian distribution is the “right” one to use in a given situation.

  1. The central limit theorem says that, in the limit, if we sum an increasing number of independent random variables, the distribution approaches Gaussian

  2. It can be proven (TODO:ref) that the Gaussian distribution has the maximum entropy subject to a given value for the first and second moments. In other words, for a given mean and variance, it makes the least assumptions about the other moments.

Exercise: derive the formula for Gaussian entropy

Because of mathjax bug

Kinematics

✎🔗

Harshit

Kinematics is the study of position, velocity and acceleration of geometric points.

A point has no dimensions. For example, the center of mass of a body is a point.

Because of mathjax bug

Dynamics

✎🔗

Harshit

Because of mathjax bug

Coordinate systems

✎🔗

Reference frames

✎🔗

Required Reading: The following assumes a working familiarity with 2D and 3D Cartesian coordinate systems. If you are not familiar with Cartesian coordinate systems, please read the chapter on coordinate systems.

Time series

✎🔗

Knowledge of k:tuple

Knowledge of time series.

Knowledge of time series operations, such as upsampling and downsampling.

Transformations

✎🔗

Required Reading: The following assumes a working familiarity with 2D and 3D Cartesian reference frames. If you are not familiar with Cartesian reference frames, please read the chapter on reference frames. Some familiarity with linear alegra is also helpful.

Types

✎🔗

Introduction to type theory.

Knowledge of set theory.

Computer science concepts

✎🔗

This unit will contain brief explanations / pointers to basic concepts of computer science.

Real-time Operating system

✎🔗

to write

Because of mathjax bug

A course in autonomy

✎🔗

These are the theory units.

Because of mathjax bug

Autonomous Vehicles

✎🔗

Liam

Autonomous Vehicles in the News

✎🔗

These days it is hard to separate the fact from the fiction when it comes to autonomous vehicles, particularly self-driving cars. Virtually every major car manufacturer has pledged to deploy some form of self-driving technology in the next five years. In addition, there are many startups and software companies which are also known to be developing self-driving car technology.

Here’s a non-exhaustive list of some of companies that are actively developing autonomous cars:

Levels of Autonomy

✎🔗

Before even discussing any detailed notion of autonomy, we have to specify exactly what we are talking about. In the United States, the governing body is the NHTSA, and they have recently (Oct 2016) redefined the so-called “levels of autonomy” for self-driving vehicles.

In broad terms, they are as follows

  • Level 0: the human driver does everything;
  • Level 1: an automated system on the vehicle can sometimes assist the human driver conduct some parts of the driving task;
  • Level 2: an automated system on the vehicle can actually conduct some parts of the driving task, while the human continues to monitor the driving environment and performs the rest of the driving task;
  • Level 3: an automated system can both actually conduct some parts of the driving task and monitor the driving environment in some instances, but the human driver must be ready to take back control when the automated system requests;
  • Level 4: an automated system can conduct the driving task and monitor the driving environment, and the human need not take back control, but the automated system can operate only in certain environments and under certain conditions; and
  • Level 5: the automated system can perform all driving tasks, under all conditions that a human driver could perform them.
Because of mathjax bug

Autonomy overview

✎🔗

Liam

This unit introduces some basic concepts ubiquitous in autonomous vehicle navigation.

Basic Building Blocks of Autonomy

✎🔗

The minimal basic backbone processing pipeline for autonomous vehicle navigation is shown in Figure 2.1.

The basic building blocks of any autonomous vehicle

For an autonomous vehicle to function, it must achieve some level of performance for all of these components. The level of performance required depends on the task and the required performance. In the remainder of this section, we will discuss some of the most basic options. In the next section we will briefly introduce some of the more advanced options that are used in state-of-the-art autonomous vehicles.

Sensors

✎🔗
Velodyne 3D Laser Scanner
Camera
Automotive Radar
GPS Receiver
Inertial Measurement Unit
Some common sensors used for autonomous navigation

Sensor A sensor is a device that or mechanism that is capable of generating a measurement of some external phyiscal quantity

In general, sensors have two major types. Passive sensors generate measurements without affecting the environment that they are measuring. Examples include inertial sensors, odometers, GPS receivers, and cameras. Active sensors emit some form of energy into the environment in order to make a measurement. Examples of this type of sensor include Light Detection And Ranging (LiDAR), Radio Detection And Ranging (RaDAR), and Sound Navigation and Ranging (SoNAR). All of these sensors emit energy (from different spectra) into the environment and then detect some property of the energy that is reflected from the environment (e.g., the time of flight or the phase shift of the signal)

Raw Data Processing

✎🔗

The raw data that is input from a sensor needs to be processed in order to become useful and even understandandable to a human.

First, calibration is usually required to convert units, for example from a voltage to a physical quantity. As a simple example consider a thermometer, which measures temperature via an expanding liquid (usually mercury). The calibration is the known mapping from amount of expansion of liquid to temperature. In this case it is a linear mapping and is used to put the markings on the thermometer that make it useful as a sensor.

We will distiguish between two fundamentally types of calibrations.

Intrinsic Calibration An intrinsic calibration is required to determine sensor-specific paramaters that are internal to a specific sensor.

Extrinsic Calibration An extrinsic calibration is required to determine the external configuration of the sensor with respect to some reference frame.

For more information about reference frames check out Unit G-20 - Reference frames

Calibration is very important consideration in robotics. In the field, the most advanced algorithms will fail if sensors are not properly calibrated.

Once we have properly calibrated data in some meaningful units, we often do some preprocessing to reduce the overall size of the data. This is true particularly for sensors that generate a lot of data, like cameras. Rather than deal with every pixel value generated by the camera, we will process an image to generate feature-points of interest. In “classical” computer vision many different feature descriptors have been proposed (Harris, BRIEF, BRISK, SURF, SIFT, etc), and more recently Convolutional Neural Networks (CNNs) are being used to learn these features.

The important property of these features is that they should be as easily to associate as possible across frames. In order to achieve this, the feature descriptors should be invariant to nuissance parameters.

Top: A raw image with feature points indicated. Bottom: Lines projected onto ground plane using extrinsic calibration and ground projections

State Estimation

✎🔗

Now that we have used our sensors to generate a set of meaningful measurements, we need to combine these measurements together to produce an estimate of the underlying hidden state of the robot and possibly to environment.

State The state $\state_t \in \statesp$ is a sufficient statistic of the environment, i.e. it contains all sufficient information required for the robot to carry out its task in that environment. This can (and usually does) include the configuration of the robot itself.

What variables are maintained in the statespace $\statesp$ depends on the problem at hand. For example we may just be interested in a single robot’s configuration in the plane, in which case $\state_t \equiv \pose_t$. However, in other cases, such as simultaneous localization and mapping, me may also be tracking the map in the state space.

According to Bayesian principles, any system parameters that are not fully known and deterministic should be maintained in the state space.

In general, we do not have direct access to values in $\state$, instead we rely on our (noisy) sensor measurements to tell us something about them, and then we infer the values.

The video is at https://vimeo.com/232324847.

Lane Following in Duckietown. *Top Right*: Raw image; *Bottom Right*: Line detections; *Top Left*: Line projections and estimate of robot position within the lane (green arrow); *Bottom Left*: Control signals sent to wheels.

The animation in Figure 2.4 shows the lane following procedure. The output of the state estimator produces the green arrow in the top left pane.

Actuation

✎🔗

The very innermost control loop deals with actually tracking the correct voltage to be sent to the motors. This is generally executed as close to the hardware level as possible. For example we have a Stepper Motor HAT See the parts list.

Infrastructure and Prior Information

✎🔗

In general, we can make the autonomous navigation a simpler one by exploiting existing structure, infastructure, and contextual prior knowledge.

Infrastructure example: Maps or GPS satellites

Structure example: Known color and location of lane markings

Contextual prior knowledge example: Cars tend to follow the Rules of the Road

Advanced Building Blocks of Autonomy

✎🔗

The basic building blocks enable static navigation in Duckietown. However, many other components are necessary for more realistic scenarios.

Object Detection

✎🔗
Advanced Autonomy: Object Detection

One key requirement is the ability to detect objects in the world such as but not limited to: signs, other robots, people, etc.

SLAM

✎🔗

The simultaneous localization and mapping (SLAM) problem involves simultaneously estimating not only the robot state but also the map at the same time, and is a fundamental capability for mobile robotics. In autonomous driving, generally the most common application for SLAM is actual in the map-building task. Once a map is built then it can be pre-loaded and then used for pure localization. A demonstration of this in Duckietown is shown in Figure 2.9.

The video is at https://vimeo.com/232333888.

Localization in Duckietown

Modern Robotic Systems

✎🔗

Andrea Censi

The many parts of a robotic system

Modern robotics development

Example of cloud learning, annotators in the loop scenario.

System architectures basics

✎🔗

Andrea Censi

Physical and logical architectures.

Deployment as mapping a physical architecture onto the logical.

Basic graph concepts.

Basic operating systems concepts.

Logical and physical architecture

✎🔗

When we design a robotic system, or any cyber-physical system, we distinguish between what we shall call “logical architecture” and the “physical architecture”.

The logical architecture describes how the functionality is divided in abstract modules, and what these modules communicate.

The physical architecture describes how the modules are instantiated. For example, this includes the information of how many processors are used, and which processor runs which routine.

Logical architecture

✎🔗

The logical architecture is independent of the implementation (the hardware, the language.)

The logical architecture describes:

  • The system decomposition in components
  • The data flow: who tells whom what
  • How knows what: other sources of information, such as priors.
  • How information is represented

A logical architecture would also describe what are the representations used. This is explored more fully in the unit about representations.

Physical architecture

✎🔗

Processors

Buses / networks

Middleware

Orchestrators

In ROS that is the roscore program.

Autonomy architectures

✎🔗

Andrea Censi

Representations

✎🔗

Matt

Required Reading: The following assumes working knowledge of 2D and 3D Cartesian coordinate systems, reference frames, and coordinate transformations. If you are not familiar with these topics, please see the following preliminary chapters.

Coordinate systems

Reference frames

Coordinate transformations.

Robots are embodied autonomous agents that interact with a physical environment.

As you read through this book, you will find that shared representations of the agent (i.e., the robot) and the environment in which it operates are fundamental to a robot’s ability to sense, plan, and act—the capabilities that are key to making a robot a robot.

The state $\state_t \in \statesp$ is a representation that consists of a compilation of all knowledge about the robot and its environment that is sufficient both to perform a particular task as well as to predict the future. Of course, “predict the future” is vague, and we can not expect the state to include knowledge to predict everything about the future, but rather what is relevant in the context of the task.

The state $\state_t \in \statesp$ is a representation of the robot and the environment that is sufficient to predict the future in the context of the task being performed.

comment

I understand that explicitly referencing the task as relevant to the notion of state is odd, but I want to convey that the representation that we choose for the state does depend upon the task.

Now, let’s be a bit more formal with regards to what we mean by a state being “sufficient” to predict the future. We are specifically referring to a state that exhibits the Markov property, i.e., that it provides a complete summary of the past (again, in the context of the task). Mathematically, we say that a state is Markov if the future state is independent of the past given the present state (technically, this corresponds to first-order Markov):

\begin{equation} p(x_{t+1} \vert x_t, a_t, x_{t-1}, a_{t-1}, \ldots, x_0, a_0) = p(x_{t+1} \vert x_t, a_t) \end{equation}

A state exhibits the Markov property if and only if the above holds.

comment

I usually say that “state” is all that is needed to predict the future. The agent only needs to keep track of a smaller thing than the state to act optimally. There isn’t a good name to use; but it should be distinct from “state”.

comment

I think that this oversimplifies things. What is it about the future that is being predicted? Certainly, the states used by autonomous systems aren’t sufficient to predict everything about the future (e.g., a self-driving car can’t predict whether a street light is going to come on, but that probably doesn’t matter).

Knowledge about the robot and its environment is often extracted from the robot’s multimodal sensor streams, such as wheel encoders and cameras. Consequently, one might choose to formulate the state as the collection of all of the measurements that the robot acquires over time. Indeed, the use of low-level sensor measurements as the representation of the state has a long history in robotics and artificial intelligence and has received renewed attention of-late, notably in the context of deep learning approaches to visuomotor policy learning.

However, while measurement history is a sufficient representation of the robot and its operating environment, it serves as a challenging definition of state.

First, raw measurements are redundant, both within a single observation and across successive measurements. For example, one doesn’t need to reason over all pixels in an image, let alone all pixels within successive images to understand the location of a street sign.

Second, raw measurements contain a large amount of information that is not necessary for a given task. For example, the pixel intensities that capture the complex diffusion light due to clouds convey information that is not useful for self-driving cars. Requiring algorithms that deal with state to reason over these pixels would be unnecessarily burdensome.

Third, measurement history is very inefficient: its size grows linearly with time as the robot makes new observations, and it may be computationally intractable to access and process such a large amount of data.

This motivates the desire for a minimal representation that expresses knowledge that is both necessary and sufficient for the robot to perform a given task. More concretely, we will consider parameterized (symbolic) formulations of the state and will prefer representations that involve as few parameters as possible, subject to the state being Markov and the constraints imposed by the task.

Metric spaces (namely, Euclidean spaces) constitute the most commonly adopted state space in robotics, be it through the use of feature-based parameterizations of the state or gridmap representation. However, it is not uncommon to define the state of the robot and its environment in terms of a topological space or a hybrid metric-topological space.

Importantly, the exteroceptive and proprioceptive sensor measurements from which the robot’s perception algorithms infer the state are noisy, the models that describe the motion of the robot and environment are error-prone, and many aspects of the state are not directly observable (e.g., your Duckiebot may not be able to “see” some of the other Duckiebots on the road due to occlusions or the cameras limited field-of-view). As a result, robots must reason over probabilistic models of the state, commonly referred to as the belief, in order to effectively mitigate this uncertainty.

Modern signal processing

✎🔗

Andrea Censi

Understanding of time series. See: Unit G-21 - Time series.

Understanding of the basic concepts of event-based processing and why it is different from periodic processing.

Understanding of latency, frequency.

Middleware

✎🔗

Andrea

Knowledge of middleware, marshalling, service discovery.

Modularization and Contracts

✎🔗

Andrea Censi

Configuration management

✎🔗

Andrea

How to deal with configuration effectively.

Duckiebot modeling

✎🔗

Obtaining a mathematical model of the Duckiebot is important in order to (a) understand its behavior and (b) design a controller to obtain desired behaviors and performances, robustly.

The Duckiebot uses an actuator (DC motor) for each wheel. By applying different torques to the wheels a Duckiebot can turn, go straight (same torque to both wheels) or stay still (no torque to both wheels). This driving configuration is referred to as differential drive.

In this section we will derive the model of a differential drive wheeled robot. The Duckiebot model will receive voltages as input (to the DC motors) and produce a configuration, or pose, as output. The pose describes unequivocally the position and orientation of the Duckiebot with respect to some Newtonian “world” frame.

Different methods can be followed to obtain the Duckiebot model, namely the Lagrangian or Newton-Euler, we choose to describe the latter as it arguably provides a clearer physical insight. Showing the equivalence of these formulations is an interesting exercises that the interested reader can take as a challenge. A useful resource for modeling of a Duckiebot may be found here [27].

Requires:k:reference_frames (inertial, body), k:intro-transformations (Cartesian, polar)

k:intro-kinematics

k:intro-dynamics

Suggested: k:intro-ODEs-to-LTIsys

k:diff-drive-robot-model

Preliminaries

✎🔗

relabel inertial frame -> local frame; $(\cdot)^I \rightarrow (\cdot)^L$

We first briefly recapitulate on the (reference frames)[#reference-frames] that we will use to model the Duckiebot, with the intent of introducing the notation used throughout this chapter. It is important to note that we restrict the current analysis to the plane, so all of the following in defined in $\reals^2$.

To describe the behavior of a Duckiebot three reference frames will be used:

  • A “world” frame: a right handed fixed reference system with origin in some arbitrary point $O$. We will indicate variables expressed in this frame with a superscript $W$, e.g., $X^W$, unless there is no risk of ambiguity, in which case no superscript will be used.

  • An “inertial” frame: a fixed reference system parallel to the “world” frame, that spans the plane on which the Duckiebot moves. We will denote its axis as $\{X_I, Y_I\}$, and it will have origin in point $A=(x_A, y_A)$, i.e., the midpoint of the robot’s wheel axle. We will indicate variables expressed in this frame with a superscript $I$, e.g., $X^I$, unless there is no risk of ambiguity, in which case no superscript will be used.

  • A body (or “robot”) frame: a local reference frame fixed with respect to the robot, centered in $A$ as well. The $x$ axis points in the direction of the front of the robot, and the $y$ axis lies along the axis between the wheels, so to form a right handed reference system. We denote the robot body frame with $\{X_R, X_R\}$. The same superscript convention as above will be used. The wheels will have radius $R$.

The robot is assumed to be a rigid body, symmetric, and $X_r$ coincides with axis of symmetry. Moreover, the wheels are considered identical and at the same distance, $L$, from the axle midpoint $A$.

Moreover:

  • The center of mass $C^W = (x_c, y_c)$ of the robot is on the $x_r$ axis, at a distance $c$ from $A$, i.e., ($C^R = (c, 0)$);

  • $X^r$ forms an orientation angle $\theta$ with the local horizontal.

These notations are summarized in Figure 11.1.

Relevant notations for modeling a differential drive robot

Kinematics

✎🔗

In this section we derive the kinematic model of a differential drive mobile platform under the assumptions of (a) no lateral slipping and (b) pure rolling of the wheels. We refer to these two assumptions as kinematic constraints.

Differential drive robot kinematic constraints

✎🔗

The kinematic constraints are derived from two assumptions:

  • No lateral slipping motion: the robot cannot move sideways, but only in the direction of motion, i.e., its lateral velocity in the robot frame is zero: $$ \label{eq:mod-no-lat-slip-constraint-r}\tag{4} \dot y_A^r = 0. $$

By inverting \eqref{eq:mod-rotation-r2i}, this constraint can be expressed through the inertial frame variables, yielding:

$$ \label{eq:mod-no-lat-slip-constraint-i}\tag{5} \dot y_A(t) \cos \theta(t) -\dot x_A(t) \sin \theta(t) = 0. $$

Imposing \eqref{eq:mod-no-lat-slip-constraint-i} on \eqref{eq:mod-A-dot-polar} results in:

$$ \label{eq:mod-no-lat-slip-constraint-v_w}\tag{6} v_w(t) = \dot y_C^I(t) \cos\theta(t) - \dot x_C^I(t) \sin\theta(t), $$

and by recalling that $C^R = (c,0)$:

$$ \label{eq:mod-no-lat-slip-constraint-C}\tag{7} \dot y_C^I(t) \cos\theta(t) - \dot x_C^I(t) \sin\theta(t) = c\dot\theta(t). $$

Hence, we obtain the strongest expression of this constraint:

$$ \label{eq:mod-no-lat-slip-final}\tag{8} v_w(t) = c\dot\theta(t), $$

and therefore:

$$ \label{eq:mod-no-lat-slip-final-dot}\tag{9} \dot v_w(t) = c\ddot\theta(t). $$

a simpler way of deriving \eqref{eq:mod-no-lat-slip-final-dot} is noticing, from \eqref{eq:mod-v_A^R}, that $\dot y_A^R = v_w(t) - c\dot\theta(t)$.

  • Pure rolling: the wheels never slips or skids (Figure 11.3). Recalling that $R$ is the radius of the wheels (identical) and letting $\dot \varphi_{l}, \dot \varphi_{r}$ be the angular velocities of the left and right wheels respectively, the velocity of the ground contact point P in the robot frame is given by:
Pure rolling kinematic constraint

\begin{align} \label{eq:mod-pure-rolling}\tag{26} \left\{ \begin{array}{ll} v_{P,r} &= R \dot \varphi_{r} \\ v_{P,l} &= R \dot \varphi_{l} \end{array} \right.. \end{align}

Another notable consequence of this assumption is that, always in the robot frame, the full power of the motor can be translated into a propelling force for the vehicle in the longitudinal direction. Or, more simply, it allows to write:

$$ \label{eq:mod-force-and-torque}\tag{10} F_{u, (\cdot)}(t)R = \tau_{(\cdot)}(t), $$

where $\tau_{(\cdot)}(t)$ is the torque exerted by each motor on its wheel $(\cdot) = {l,r}$.

Differential drive robot kinematic model

✎🔗

In a differential drive robot, controlling the wheels at different speeds generates a rolling motion of rate $\omega = \dot \theta$. In a rotating field there always is a fixed point, the center of instantaneous curvature (ICC), and all points at distance $d$ from it will have a velocity given by $\omega d$, and direction orthogonal to that of the line connecting the ICC and the wheels (i.e., the axle). Therefore, by looking at Figure 11.1, we can write:

\begin{align} \label{eq:mod-kin-1}\tag{27} \left\{ \begin{array}{l} \dot \theta (d-L) &= v_l \\ \dot \theta (d+L) &= v_r \end{array} \right., \end{align}

from which:

\begin{align} \label{eq:mod-kin-2}\tag{28} \left\{ \begin{array}{l} d &= L \frac{v_r + v_l}{v_r - v_l} \\ \dot \theta &= \frac{v_r - v_l}{2L} \end{array} \right.. \end{align}

A few observations stem from \eqref{eq:mod-kin-2}:

  • If $v_r = v_l$ the bot does not turn ($\dot \theta = 0$), hence the ICC is not defined;
  • If $v_r = - v_l$, then the robot “turns on itself”, i.e., $d=0$ and $ICC \equiv A$;
  • If $v_r = 0$ (or $v_l = 0$), the rotation happens around the right (left) wheel and $d = 2L$ ($d = L$).

Moreover, a differential drive robot cannot move in the direction of the ICC, it is a singularity.

By recalling the no lateral slipping motion \eqref{eq:mod-no-lat-slip-constraint-r} hypothesis and the pure rolling constraint \eqref{eq:mod-pure-rolling}, and noticing that the translational velocity of $A$ in the robot frame is $v_A = \dot \theta d = (v_r+v_l)/2$ we can write:

\begin{align} \label{eq:mod-kin-3}\tag{29} \left\{ \begin{array}{l} \dot x_A^R &= R (\dot \varphi_R +\dot \varphi_L)/2 \\ \dot y_A^R &= 0 \\ \dot \theta &= \omega = R(\dot \varphi_R - \dot \varphi_L)/(2L) \end{array} \right., \end{align}

which in more compact yields the simplified forward kinematics in the robot frame:

\begin{align} \label{eq:mod-forward-kinematics-robot-frame}\tag{30} \left[ \begin{array}{c} \dot x_A^R \\ \dot y_A^R \\ \dot \theta \end{array} \right] = \left[ \begin{array}{cc} \frac{R}{2} & \frac{R}{2} \\ 0 & 0 \\ \frac{R}{2L} & -\frac{R}{2L} \\ \end{array} \right] \left[ \begin{array}{c} \dot \varphi_R \\ \dot \varphi_L \end{array} \right]. \end{align}

Finally, by using \eqref{eq:mod-rot-mat}, we can recast \eqref{eq:mod-forward-kinematics-robot-frame} in the inertial frame.

The simplified forward kinematics model of a differential drive vehicle is given by: \begin{align} \label{eq:mod-forward-kinematics-inertial-frame}\tag{31} \displaystyle \avec{\dot q}^I = \amat{R}(\theta) \left[ \begin{array}{c} \dot x_A^r \\ \dot y_A^r \\ \dot \theta \end{array} \right] = \left[ \begin{array}{cc} \frac{R}{2} \cos \theta & \frac{R}{2} \cos \theta \\ \frac{R}{2} \sin \theta & \frac{R}{2} \sin \theta \\ \frac{R}{2L} & -\frac{R}{2L} \\ \end{array} \right] \left[ \begin{array}{c} \dot \varphi_R \\ \dot \varphi_L \end{array} \right] = \left[ \begin{array}{cc} \cos \theta & 0 \\ \sin \theta & 0 \\ 0 & 1 \\ \end{array} \right] \left[ \begin{array}{c} v_A \\ \omega \end{array} \right]. \end{align}

Simplified dynamic model

✎🔗

By implementing the kinematic constraints formulations derived above, i.e., the no lateral slipping (\eqref{eq:mod-no-lat-slip-final-dot}) and pure rolling (\eqref{eq:mod-force-and-torque}) in the general dynamic model, it is straightforward to obtain:

\begin{align} \label{eq:mod-dyn-model-a}\tag{32} \dot{v}_u (t) &= c \dot \theta^2(t) + \frac{1}{RM} (\tau_R(t)+\tau_L(t)) \\ \dot v_w(t) &= c \dot\theta(t) \\ \ddot \theta &= - \frac{Mc}{Mc^2+J} \dot\theta(t)v_u(t) + \frac{L}{R(Mc^2+J)}(\tau_R(t)-\tau_L(t)) \end{align}

DC motor dynamic model

✎🔗

The equations governing the behavior of a DC motor are driven by an input armature voltage $V(t)$:

\begin{align} V(t) &= Ri(t) + L \frac{di}{dt} + e(t) \\ e(t) &= K_b \dot\varphi(t) \\ \tau_m(t) &= K_t i(t) \\ \tau(t) &= N \tau_m(t), \end{align}

where $(K_b, K_t)$ are the back emf and torque constants respectively and $N$ is the gear ratio ($N=1$ in the Duckiebot).

Figure 11.4 shows a diagram of a typical DC motor.

Diagram of a DC motor

Having a relation between the applied voltage and torque, in addition to the dynamic and kinematic models of a differential drive robot, allows us to determine all possible state variables of interest.

torque disturbances acting on the wheels, such as the effects of friction, can be modeled as additive terms (of sign opposite to $\tau$) in the DC motor equations.

Conclusions

✎🔗

In this chapter we derived a model for a differential drive robot. Although several simplifying assumption were made, e.g., rigid body motion, symmetry, pure rolling and no lateral slipping - still the model is nonlinear.

Regardless, we now have a sequence of descriptive tools that receive as input the voltage signal sent by the controller, and produce as output any of the state variables, e.g., the position, velocity and orientation of the robot with respect to a fixed inertial frame.

Several outstanding questions remain. For example, we need to determine what is the best representation for our robotic platform - polar coordinates, Cartesian with respect to an arbitrary reference point? Or maybe there is a better choice?

Finally, the above model assumes the knowledge of a number of constants that are characteristic of the robot’s geometry, materials, and the DC motors. Without the knowledge of those constant the model could be completely off. Determination of these parameters in a measurement driven way, i.e., the “system identification” of the robot’s plant, is subject of the odometry class.

Because of mathjax bug

Odometry Calibration

✎🔗

Jacopo

Because of mathjax bug

Computer vision basics

✎🔗

Matt

For the interested reader, there are several excellent books that discuss fundamental topics in computer vision. These include (in no particular order), Computer Vision: Algorithims and Applications (available online), Computer Vision: A Modern Approach and Multiple View Geometry in Computer Vision.

✎

Topics:

  • Objective: Discover what is present in the world, where things are, and what actions are taking place from image sequences. In the context of robotics, this corresponds to learning a (probabilistic) world model that encodes the robot’s environment, i.e., building a representation of the environment.
  • History of computer vision
    • Summer Vision Project
    • OCR (license plates, signs, checks, mail, etc.)
    • Face detection
    • Structure from motion
    • Dense reconstruction
    • Augmented reality
    • Applications to self-driving cars and beyond
Because of mathjax bug

Camera geometry

✎🔗

Matt

We could break this up into separate sub-sections for (prospective) projection and lenses.

Topics:

  • Pinhole cameras (tie into eclipse viewing, accidental cameras)
  • Geometry of projection: equations of projection; vanishing points; weak perspective; orthographic projection;
  • Review of 3D transformations (translation and rotation); homogeneous transformations
  • Perspective projection in homogeneous coordinates
✎
  • Lenses: Why?; first-order optics (thin lens); thick lens
  • First-order optics (thin lens)
Because of mathjax bug

Camera calibration

✎🔗

Matt

The discussion of intrinsic and extrinsic models could be moved up to the geometry subsection

Topics

  • Why calibration?
  • Intrinsic parameters: idealized perspective projection; pixel scaling (square and non-square); offset; skew.
  • Intrinsic model expressed in homogeneous coordinates
  • Extrinsic parameters: translation and rotation of camera frame (non-homogeneous and homogeneous)
  • Combined expression for intrinsic and extrinsic
  • Calibration targets
  • Calibration models
Because of mathjax bug

Image filtering

✎🔗

Matt

Background: Discrete-time signal processing

We currently don’t have a preliminary section on signal processing and it’s probably sufficient to point readers to a good reference (e.g., Oppenheim and Schafer).

Topics

May want to stick to linear filtering

  • Motivation (example)
  • Convolution: definition and properties (including differentiation)
  • Linear filters: examples (sharpening and smoothing)
  • Gaussian filters
    • kernels
    • properties (convolution with a Gaussian, separability)
Because of mathjax bug

Illumination invariance

✎🔗

Matt

Because of mathjax bug

Line Detection

✎🔗

Matt

Topics:

  • What is edge detection?
  • Image derivatives
  • Finite difference filters
  • Image gradients
  • Effects of noise and need for smoothing
  • Derivative theorem of convolution
  • Derivative of Gaussian filters
  • Smoothing vs. derivative filters
  • Line detection as an edge detection problem
Because of mathjax bug

Feature extraction

✎🔗

Matt

Because of mathjax bug

Place recognition

✎🔗

Matt

Because of mathjax bug

Filtering 1

✎🔗

Liam

Markov Property

Bayes Filter

Graphical representation

Kalman Filter

Extended Kalman Filter

Because of mathjax bug

Filtering 2

✎🔗

Liam

MAP estimation

Laplace Approximation

Cholesky Decomposition?

Incrementalization - Givens Rotations - iSAM

Because of mathjax bug

Mission planning

✎🔗

ETH

Because of mathjax bug

Planning in discrete domains

✎🔗

ETH

Because of mathjax bug

Motion planning

✎🔗

ETH

Because of mathjax bug

RRT

✎🔗

ETH

Because of mathjax bug

Feedback control

✎🔗

Jacopo

Because of mathjax bug

PID Control

✎🔗

Jacopo

Because of mathjax bug

MPC Control

✎🔗

Jacopo

Because of mathjax bug

Object detection

✎🔗

Nick and David

Because of mathjax bug

Object classification

✎🔗

Nick and David

Because of mathjax bug

Object tracking

✎🔗

Nick and David

Because of mathjax bug

Reacting to obstacles

✎🔗

Jacopo

Because of mathjax bug

Semantic segmentation

✎🔗

Nick and David

Because of mathjax bug

Text recognition

✎🔗

Nick

Because of mathjax bug

SLAM - Problem formulation

✎🔗

Liam

Because of mathjax bug

SLAM - Broad categories

✎🔗

Liam

Because of mathjax bug

VINS

✎🔗

Liam

Because of mathjax bug

Advanced place recognition

✎🔗

Liam

Because of mathjax bug

Fleet level planning (placeholder)

✎🔗

ETH

Because of mathjax bug

Fleet level planning (placeholder)

✎🔗

ETH

Because of mathjax bug

Bibliography

✎🔗
Liam Paull, Jacopo Tani, Heejin Ahn, Javier Alonso-Mora, Luca Carlone, Michal Cap, Yu Fan Chen, Changhyun Choi, Jeff Dusek, Daniel Hoehener, Shih-Yuan Liu, Michael Novitzky, Igor Franzoni Okuyama, Jason Pazis, Guy Rosman, Valerio Varricchio, Hsueh-Cheng Wang, Dmitry Yershov, Hang Zhao, Michael Benjamin, Christopher Carr, Maria Zuber, Sertac Karaman, Emilio Frazzoli, Domitilla Del Vecchio, Daniela Rus, Jonathan How, John Leonard, and Andrea Censi. Duckietown: an open, inexpensive and flexible platform for autonomy education and research. In IEEE International Conference on Robotics and Automation (ICRA). Singapore, May 2017. pdf supp. materialbibtex Bruno Siciliano and Oussama Khatib. Springer Handbook of Robotics. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2007.    Jacopo Tani, Liam Paull, Maria Zuber, Daniela Rus, Jonathan How, John Leonard, and Andrea Censi. Duckietown: an innovative way to teach autonomy. In EduRobotics 2016. Athens, Greece, December 2016. pdf supp. materialbibtex Tosini, G., Ferguson, I., Tsubota, K. Effects of blue light on the circadian system and eye physiology. Molecular Vision, 22, 61–72, 2016 (online). Albert Einstein. Zur Elektrodynamik bewegter Körper. (German) On the electrodynamics of moving bodies. Annalen der Physik, 322(10):891–921, 1905.    DOI  Ivan Savov. Linear algebra explained in four pages. https://minireference.com/static/tutorials/linear_algebra_in_4_pages.pdf, 2017. Online; accessed 23 August 2017.    Kaare Brandt Petersen and Michael Syskind Pedersen. The matrix cookbook.    .pdf  Matrix (mathematics). Matrix (mathematics) — Wikipedia, the free encyclopedia, 2017. Online; accessed 2-September-2017.    www:  Peter D. Lax. Functional Analysis. Wiley Interscience, New York, NY, USA, 2002.    Athanasios Papoulis and S. Unnikrishna Pillai. Probability, Random Variables and Stochastic Processes. McGraw Hill, fourth edition, 2002.    David J. C. MacKay. Information Theory, Inference & Learning Algorithms. Cambridge University Press, New York, NY, USA, 2002.    H. Choset, W. Burgard, S. Hutchinson, G. Kantor, L. E. Kavraki, K. Lynch, and S. Thrun. Principles of robot motion: Theory, agorithms, and implementation. MIT Press, June 2005.    Steven M. LaValle. Planning Algorithms. Cambridge University Press, New York, NY, USA, 2006.    M. Miskowicz. Event-Based Control and Signal Processing. Embedded Systems. CRC Press, 2015.    http  Karl J. Aström. Event Based Control, pages 127–147. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.    DOI  http 
Summary. In spite of the success of traditional sampled-data theory in computer control it has some disadvantages particularly for distributed, asynchronous, and multi-rate system. Event based sampling is an alternative which is explored in this paper. A simple example illustrates the differences between periodic and event based sampling. The architecture of a general structure for event based control is presented. The key elements are an event detector, an observer, and a control signal generator, which can be regarded as a generalized data-hold. Relations to nonlinear systems are discussed. Design of an event based controller is illustrated for a simple model of a micro-mechanical accelerometer.

Edward A Lee and David G Messerschmitt. Synchronous data flow. Proceedings of the IEEE, 75(9):1235–1245, 1987.    Rajit Manohar and K Mani Chandy. Delta-dataflow networks for event stream processing. pages 1–6, June 2010.    Romano M DeSantis. Modeling and path-tracking control of a mobile wheeled robot with a differential drive. Robotica, 13(4):401–410, 1995.    Richard Szeliski. Computer Vision: Algorithms an Applications. Springer, 2010.    http  David Forsyth and Jean Ponce. Computer Vision: A Modern Approach. Prentce Hall, Upper Saddle River, NJ, 2011.    Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, second edition, 2004.    Censi Tani. Project pitches. Slides, 10 2017.    Davide Scaramuzza. Design and realization of a stereoscopic vision system for robotics, with applications to tracking of moving objects and self-localization. Master thesis, department of electronic and information engineering, University of Perugia, Perugia, Italy, 2005.    .pdf  Gang Yi Jiang. Lane and obstacle detection based on fast inverse perspective mapping algorithm. In IEEE Xplore, Conference: Systems, Man, and Cybernetics, 2000 IEEE International Conference on, Volume: 4, Febuary 2000.    http  Alessandra Fascioli Massimo Bertozzi, Alberto Broggi. Stereo inverse perspective mapping: Theory and applications. Image and Vision Computing 16 (1998), 1997.    .pdf  Kesheng Wu. Optimizing connected component labeling algorithms, 2005.    http  Richard Fitzpatrick. Moment of inertia tensor, 03 2011.    .html  Google offers raspberry pi owners this new ai vision kit to spot cats, people, emotions, 12 2017.    http  Davide Scaramuzza. Lecture: Vision algorithms for mobile robotics, 2017.    .html  Davide Scaramuzza. Towards robust and safe autonomous drones, September 2015.    http  Rasmussen. Presentation on theme: "computer vision : Cisc 4/689".    http  Inverse perspective mapping, September 2012.    http  VIKAS GUPTA. Color spaces in opencv (c++ / python), May 2017.    http  Convert from hsv to rgb color space, 2018.    http  Michel Alves. About perception and hue histograms in hsv space, July 2015.    http  CHENG. Slides from the lecture "computer graphics".    Hsl and hsv, 12 2017.    http 
Because of mathjax bug

Exercises

✎🔗

These exercises can guide you from the status of a novice coder to experienced roboticist.

Because of mathjax bug

Exercise: Basic image operations

✎🔗

Andrea Daniele

Exercise: Basic image operations, adult version

✎🔗

Andrea Daniele

dt-image-flip specification

✎🔗

The program image-ops expects exactly two arguments: a filename (a JPG file) and a directory name.

$ dt-image-flip file outdir

If the file does not exist, the script must exit with error code 2.

If the file cannot be decoded, the script must exit with error code 3.

If the file exists, then the script must create:

  • outdir/regular.jpg: a copy of the initial file
  • outdir/flip.jpg: the file, flipped vertically.
  • outdir/side-by-side.jpg: the two files, side by side.

If any other error occurs, the script should exit with error code 99.

The original picture.
The output flip.jpg
The output side-by-side.jpg
Example input-output for the program image-ops.

Testing it works with image-ops-tester

✎🔗

We provide 4 scripts that can be used to make sure that you wrote a conforming dt-image-flip. The scripts are image-ops-tester-good, image-ops-tester-bad1, image-ops-tester-bad2, and image-ops-tester-bad3. You can find them in the directory /exercises/dt-image-flip/image-ops-tester in the duckietown/duckuments repository.

The script called image-ops-tester-good tests your program in a situation in which we expect it to work properly. The 3 “bad” test scripts (i.e., image-ops-tester-bad1 through image-ops-tester-bad3) test your code in situations in which we expect your program to complain in the proper way.

Use them as follows:

$ image-ops-tester-scenario candidate-program

The tester scripts must be called from their own location. Make sure to change your working directory to /exercises/dt-image-flip/image-ops-tester before launching the tester scripts.

If the script cannot be found, image-ops-tester-scenario will return 1.

image-ops-tester-scenario will return 0 if the program exists and conforms to the specification (Section 2.3 - dt-image-flip specification).

If it can establish that the program is not good, it will return 11.

Exercise: Simple data analysis from a bag

✎🔗

Andrea Daniele

Exercise: Bag in, bag out

✎🔗

Andrea Daniele

Exercise: Bag thumbnails

✎🔗

Andrea Daniele

Exercise: Instagram filters

✎🔗

Andrea Daniele

Exercise: Bag instagram

✎🔗

Andrea Daniele

Exercise: Live Instagram

✎🔗

Andrea Daniele

Exercise: Augmented Reality

✎🔗

Jonathan Michaux and Dzenan Lapandic

Specification of dt_augmented_reality

✎🔗

In this assignment you will be writing a ROS package to perform the augmented reality exercise. The program will be invoked with the following syntax:

$ roslaunch dt_augmented_reality-robot name augmenter.launch map_file:=map file robot_name:=robot name local:=1

where map file is a YAML file containing the map (specified in Section 9.5 - Specification of the map).

If robot name is not given, it defaults to the hostname.

The program does the following:

  1. It loads the intrinsic / extrinsic calibration parameters for the given robot.
  2. It reads the map file.
  3. It listens to the image topic /robot name/camera_node/image/compressed.

  4. It reads each image, projects the map features onto the image, and then writes the resulting image to the topic

    /![robot name]/AR/![map file basename]/image/compressed

where map file basename is the basename of the file without the extension.

We provide you with ROS package template that contains the AugmentedRealityNode. By default, launching the AugmentedRealityNode should publish raw images from the camera on the new /robot name/AR/map file basename/image/compressed topic.

In order to complete this exercise, you will have to fill in the missing details of the Augmenter class by doing the following:

  1. Implement a method called process_image that undistorts raw images.
  2. Implement a method called ground2pixel that transforms points in ground coordinates (i.e. the robot reference frame) to pixels in the image.
  3. Implement a method called callback that writes the augmented image to the appropriate topic.

Specification of the map

✎🔗

The map file contains a 3D polygon, defined as a list of points and a list of segments that join those points.

The format is similar to any data structure for 3D computer graphics, with a few changes:

  1. Points are referred to by name.
  2. It is possible to specify a reference frame for each point. (This will help make this into a general tool for debugging various types of problems).

Here is an example of the file contents, hopefully self-explanatory.

The following map file describes 3 points, and two lines.

points:
    # define three named points: center, left, right
    center: [axle, [0, 0, 0]] # [reference frame, coordinates]
    left: [axle, [0.5, 0.1, 0]]
    right: [axle, [0.5, -0.1, 0]]
segments:
- points: [center, left]
  color: [rgb, [1, 0, 0]]
- points: [center, right]
  color: [rgb, [1, 0, 0]]

Exercise: Lane Filtering - Particle Filter

✎🔗

Jonathan Michaux, Liam Paull, and Miguel de la Iglesia

Exercise: Lane Filtering - Extended Kalman Filter

✎🔗

Liam Paull

Exercise: Git and conventions

✎🔗

move here the exercises we had last year about branching.

Exercise: Use our API for arguments

✎🔗

Exercise: Bouncing ball

✎🔗

Exercise: Visualizing data on image

✎🔗

Exercise: Make that into a node

✎🔗

Exercise: Instagram with EasyAlgo interface

✎🔗

Milestone: Illumination invariance (anti-instagram)

✎🔗

Make them run our code, and also visualize what’s going on

Exercise: Launch files basics

✎🔗

Exercise: Unit tests

✎🔗

Exercise: Parameters (standard ROS api)

✎🔗

Exercise: Parameters (our API)

✎🔗

Exercise: Place recognition abstraction

✎🔗

Exercise: Parallel processing

✎🔗

Exercise: Adding new test to integration tests

✎🔗

to write

Exercise: localization

✎🔗

to write

Exercise: Ducks in a row

✎🔗

to write

Exercise: Comparison of PID

✎🔗

to write

Exercise: RRT

✎🔗

to write

Because of mathjax bug

Exercise: Who watches the watchmen? (optional)

✎🔗

Software reference

✎🔗

This part describes things that you should know about UNIX/Linux environments.

Documentation writers: please make sure that every command used has a section in these chapters.

Because of mathjax bug

Ubuntu packaging with APT

✎🔗

apt-key

✎🔗

to write

apt-mark

✎🔗

to write

apt-get

✎🔗

to write

add-apt-repository

✎🔗

to write

wajig

✎🔗

to write

dpigs

✎🔗

to write

Because of mathjax bug

GNU/Linux general notions

✎🔗

Andrea

Every day Linux

✎🔗

man

✎🔗

This is an interface to the on-line reference manuals. Whenever you meet some unfamiliar commands, try use man certain_command before Googling. You will find it extremely clear, useful and self-contained.

cd

✎🔗

Go to the directory you want. If you just use:

laptop $ cd

then you will go to your home directory, i.e., ~

sudo

✎🔗

Whenever you want to modify system files, you will need sudo. Commonly touched system files including /etc, /opt and so on. Since most of you have followed the guideline to use passwordless sudo, I would recommend that make sure what you are doing with sudo before you execute the command, otherwise you may need to reinstall the system.

ls

✎🔗

List all the files and documents in the current directory. From ~/.bashrc, we know some commonly used alias. See more by man ls.

– la for ls -a which will list out all files and documents including the hidden ones(whose name starts with a dot).

– ll for ls -l which will display Unix file types, permissions, number of hard links, owner, group, size, last-modified date and filename.

cp

✎🔗

cp fileA directoryB will copy the file A to directory B. See more by executing man cp.

mkdir

✎🔗

Make new directory. See more by man mkdir.

touch

✎🔗

Update the access and modification times of the input file to current time. See more by man touch.

reboot

✎🔗

This command must be executed as root. sudo required. This will reboot your laptop or Raspberry Pi. See more by man reboot.

shutdown

✎🔗

This command requires sudo. You can set a countdown to shutdown you machine. More by man shutdown.

rm

✎🔗

Remove certain file. rm -r will remove files. More in man rm.

Users

✎🔗

passwd

✎🔗

Update password of the current user. Old password needed.

UNIX tools

✎🔗

cat

✎🔗

Cat some file will return you the content. More in man cat.

tee

✎🔗

Read from standard input and write to standard output and files. More on man tee.

truncate

✎🔗

to write

Linux disks and files

✎🔗

fdisk

✎🔗

to write

mount

✎🔗

to write

umount

✎🔗

to write

losetup

✎🔗

to write

gparted

✎🔗

to write

dd

✎🔗

to write

sync

✎🔗

to write

df

✎🔗

to write

How to make a partition

✎🔗

to write

Other administration commands

✎🔗

visudo

✎🔗

to write

update-alternatives

✎🔗

to write

udevadm

✎🔗

to write

systemctl

✎🔗

to write

Make

✎🔗

make

✎🔗

to write

Python-related tools

✎🔗

virtualenv

✎🔗

to write

pip

✎🔗

to write

Raspberry-PI commands

✎🔗

raspi-config

✎🔗

to write

vcgencmd

✎🔗

to write

raspistill

✎🔗

to write

jstest

✎🔗

to write

swapon

✎🔗

to write

mkswap

✎🔗

to write

Users and permissions

✎🔗

chmod

✎🔗

to write

groups

✎🔗

to write

adduser

✎🔗

to write

useradd

✎🔗

to write

Downloading

✎🔗

curl

✎🔗

to write

wget

✎🔗

to write

sha256sum

✎🔗

to write

xz

✎🔗

to write

Shells and environments

✎🔗

source

✎🔗

You can only do source file_name if the file can be executed by bash.

which

✎🔗

Tell you the /bin/ directory of your command. This is useful to distinguish which python you are using if you have virtualenv.

export

✎🔗

to write

Other misc commands

✎🔗

npm

✎🔗

to write

nodejs

✎🔗

to write

ntpdate

✎🔗

to write

chsh

✎🔗

to write

echo

✎🔗

to write

sh

✎🔗

to write

fc-cache

✎🔗

to write

Mounting USB drives

✎🔗

First plug in the USB drive nothing will work if you don’t do that first. Now ssh into your robot. On the command line type:

duckiebot $ lsusb

you should see your Sandisk USB drive as an entry. Congrats, you correctly plugged it in

duckiebot $ lsblk

Under name you should see sda1, with size about 28.7GB and nothing under the MOUNTPOINT column (if you see something under MOUNTPOINT congrats you are done.

Next make a directory to mount to:

duckiebot $ sudo mkdir /media/logs

Next mount the drive

duckiebot $ sudo mount -t vfat /dev/sda1 /media/logs -o umask=000

Test by running lsblk again and you should now see /media/logs under MOUNTPOINT

Unmounting a USB drive

✎🔗
duckiebot $ sudo umount /media/logs
Because of mathjax bug

Linux resources usage

✎🔗

SD Cards tools

✎🔗

Testing SD Card and disk speed

✎🔗

Test SD Card (or any disk) speed using the following commands, which write to a file called filename.

$ dd if=/dev/zero of=filename bs=500K count=1024
$ sync
$ echo 3 | sudo tee /proc/sys/vm/drop_caches
$ dd if=filename of=/dev/null bs=500K count=1024
$ rm filename

Note the sync and the echo command are very important.

Example results:

524288000 bytes (524 MB, 500 MiB) copied, 30.2087 s, 17.4 MB/s
524288000 bytes (524 MB, 500 MiB) copied, 23.3568 s, 22.4 MB/s

That is write 17.4 MB/s, read 22 MB/s.

How to burn an image to an SD card

✎🔗

A blank SD card.

An image file to burn.

An Ubuntu computer with an SD reader.

A burned image.

How to shrink an image

✎🔗

An image file to burn.

An Ubuntu computer.

A shrunk image.

Majority of content taken from here

We are going to use the tool gparted so make sure it’s installed

laptop $ sudo apt install gparted

Let the image file be image file and its name be imagename. Run the command:

laptop $ sudo fdisk -l image file

It should give you something like:

Device                       Boot  Start      End  Sectors  Size Id Type
duckiebot-RPI3-LP-aug15.img1        2048   131071   129024   63M  c W95 FAT32 (LBA)
duckiebot-RPI3-LP-aug15.img2      131072 21219327 21088256 10.1G 83 Linux

Take note of the start of the Linux partition (in our case 131072), let’s call it start. Now we are going to mount the Linux partition from the image:

laptop $ sudo losetup /dev/loop0 imagename.img -o $((start*512))

and then run gparted:

laptop $ sudo gparted /dev/loop0

In gparted click on the partition and click “Resize” under the “Partition” menu. Resize drag the arrow or enter a size that is equal to the minimum size plus 20MB

This didn’t work well for me - I had to add much more than 20MB for it to work.

Click the “Apply” check mark. Before closing the final screen click through the arrows in the dialogue box to find a line such a “resize2fs -p /dev/loop0 1410048K”. Take note of the new size of your partition. Let’s call it new size.

Now remove the loopback on the second partition and setup a loopback on the whole image and run fdisk:

laptop $ sudo losetup -d /dev/loop0
laptop $ sudo losetup /dev/loop0 image file
laptop $ sudo fdisk /dev/loop0

Command (m for help): enter d
Partition number (1,2, default 2): enter 2
Command (m for help): enter n
Partition type
p   primary (1 primary, 0 extended, 3 free)
e   extended (container for logical partitions)
Select (default p): enter p
Partition number (2-4, default 2): enter 2
First sector (131072-62521343, default 131072): start
Last sector, +sectors or +size{K,M,G,T,P} (131072-62521343, default 62521343): +new sizeK

on the last line include the + and the K as part of the size.

Created a new partition 2 of type 'Linux' and of size 10.1 GiB.

Command (m for help): enter w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Invalid argument

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

Disregard the final error.

You partition has now been resized and the partition table has been updated. Now we will remove the loopback and then truncate the end of the image file:

laptop $ fdisk -l /dev/loop0


Device       Boot  Start      End  Sectors  Size Id Type
/dev/loop0p1        2048   131071   129024   63M  c W95 FAT32 (LBA)
/dev/loop0p2      131072 21219327 21088256 10.1G 83 Linux

Note down the end of the second partition (in this case 21219327). Call this end.

laptop $ sudo losetup -d /dev/loop0
laptop $ sudo truncate -s $(((end+1)*512)) image file

You now have a shrunken image file.

It might be useful to compress it, before distribution:

laptop $ xz image file
Because of mathjax bug

Networking tools

✎🔗

Andrea

Preliminary reading:

  • Basics of networking, including

    • what are IP addresses
    • what are subnets
    • how DNS works
    • how .local names work
    • …

XXX (ref to find).

to write

Make sure that you know:

hostname

✎🔗

to write

Accessing computers using SSH

✎🔗

Andrea

Local configuration

✎🔗

The SSH configuration as a client is in the file

~/.ssh/config

Create the directory with the right permissions:

$ mkdir ~/.ssh
$ chmod 0700 ~/.ssh

Edit the file:

~/.ssh/config

(We suggest you use VIM to edit files; see a tutorial here.)

comment

laptop or duckiebot? - LP

Then add the following lines:

HostKeyAlgorithms ssh-rsa

The reason is that Paramiko, used by roslaunch, does not support the ECSDA keys.

How to login with SSH and a password

✎🔗

To log in to a remote computer remote with user remote-user, use:

$ ssh remote-user@remote

Creating an SSH keypair

✎🔗

This is a step that you will repeat twice: once on the Duckiebot, and once on your laptop.

The program will prompt you for the filename on which to save the file.

Use the convention

/home/username/.ssh/username@host name
/home/username/.ssh/username@host name.pub

where:

  • username is the current user name that you are using (ubuntu or your chosen one);
  • host name is the name of the host (the Duckiebot or laptop);

An SSH key can be generated with the command:

$ ssh-keygen -h

The session output will look something like this:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa):

At this point, tell it to choose this file:

/home/username/.ssh/username@host name

Then:

Enter passphrase (empty for no passphrase):

Press enter; you want an empty passphrase.

Enter same passphrase again:

Press enter.

Your identification has been saved in /home/username/.ssh/username@host name
Your public key has been saved in /home/username/.ssh/username@host name.pub
The key fingerprint is:
XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX username@host name
The key's randomart image is:
+--[ RSA 2048]----+
|            .    |
|       o   o  .  |
|      o = o  . o |
|       B . .  * o|
|        S o    O |
|         o o  . E|
|          o o  o |
|           o  +  |
|            .. . |
+-----------------+

Note that the program created two files.

The file that contains the private key is

/home/username/.ssh/username@host name

The file that contains the public key has extension .pub:

/home/username/.ssh/username@host name.pub

Next, tell SSH that you want to use this key.

Make sure that the file ~/.ssh/config exists:

$ touch ~/.ssh/config

Add a line containing

IdentityFile ~/.ssh/PRIVATE_KEY_FILE

(using the filename for the private key).

comment

make sure to include the full path to the file, not just the filename.

Check that the config file is correct:

$ cat ~/.ssh/config
...
IdentityFile ~/.ssh/PRIVATE_KEY_FILE
...

To copy the generated SSH key to the clipboard xclip can be used (Installation of xclip if necessary).

$ sudo apt-get install xclip
$ xclip -sel clip < ~/.ssh/username@host name.pub

How to login without a password

✎🔗

You have two computers, called “local” and “remote”, with users “local-user” and “remote-user”. Here, we assume that local and remote are complete hostnames (such as duckiebot.local.).

The two computers are on the same network and they can ping each other.

You have created a keypair for local-user on local. This procedure is described in Section 19.5 - Creating an SSH keypair.

From the local computer, local-user will be able to log in to remote computer without a password.

First, connect the two computers to the same network, and make sure that you can ping remote from local:

local $ ping remote.local

Do not continue if you cannot do this successfully.

If you have created a keypair for local-user, you will have a public key in this file on the local computer:

/home/local-user/.ssh/local-user@local.pub

This file is in the form:

ssh-rsa long list of letters and numbers local-user@local

You will have to copy the contents of this file on the remote computer, to tell it that this key is authorized.

On the local computer, run the command:

local $ ssh-copy-id remote-user@remote

now you should be able to login to the remote without a password:

local $ ssh remote-user@remote

This should succeed, and you should not be asked for a password.

Fixing SSH Permissions

✎🔗

Sometimes, SSH does not work because you have the wrong permissions on some files.

In doubt, these lines fix the permissions for your .ssh directory.

$ chmod 0700 ~/.ssh
$ chmod 0700 ~/.ssh/*

ssh-keygen

✎🔗

to write

Because of mathjax bug

Wireless networking in Linux

✎🔗

iwconfig

✎🔗

to write

iwlist

✎🔗

Moving files between computers

✎🔗

scp

✎🔗

to write

Download a file with SCP

✎🔗

Use this command:

laptop $ scp hostname:/path/to/out.jpg .

to download out.jpg to your current directory.

rsync

✎🔗

to write

Because of mathjax bug

VIM

✎🔗

Andrea

To do quick changes to files, especially when logged remotely, we suggest you use the VI editor, or more precisely, VIM (“VI iMproved”).

vi

✎🔗

to write

Atom

✎🔗

Install Atom

✎🔗

Following the instructions here:

$ sudo add-apt-repository ppa:webupd8team/atom
$ sudo apt update
$ sudo apt install atom

Using Atom to code remotely

✎🔗

With Atom, you are able to remotely code on files located on your Duckiebot with a GUI. The benefit of using Atom is that you are able to install extensions such as an IDE for Python, a Markdown previewer, or just use custom themes to avoid coding in the terminal.

Follow these instructions:

Install remote-atom

laptop $ sudo apm install remote-atom

Now, we need to edit our SSH config so that any data send to the port 52698, which is the port remote-atom is using, is forwarded via SSH to our local machine. Edit the file “~/.ssh/config”. There, you add “RemoteForward 52698 127.0.0.1:52698” to your host. The resulting host will look similar to

Host lex
    User julien
    Hostname lex.local
    RemoteForward 52698 127.0.0.1:52698

Now, we need to connect to our duckiebot via SSH and install rmate (and simultaniously rename it to ratom)

duckiebot $ sudo wget -O /usr/local/bin/ratom https://raw.github.com/aurora/rmate/master/rmate
duckiebot $ sudo chmod +x /usr/local/bin/ratom

Now, you just need to launch Atom on your local machine, go to Packages->Remote Atom->Start Server.

You can now edit a file in a terminal connected to your duckiebot via SSH by typing

duckiebot $ sudo ratom filename

And atom will automatically open the file on your local machine. In the settings of remote-atom, you can also set the package to start the server automatically on launching atom on your local machine.

Because of mathjax bug

Liclipse

✎🔗

Installing LiClipse

✎🔗

Follow the instructions at this page.

At the moment of writing, these are:

$ wget http://www.mediafire.com/file/rwc4bk3nthtxcvv/liclipse_4.1.1_linux.gtk.x86_64.tar.gz
$ tar xvzf liclipse_4.1.1_linux.gtk.x86_64.tar.gz
$ sudo ln -s `pwd`/liclipse/LiClipse /usr/bin/liclipse

Now you can run it using liclipse:

$ liclipse

When it runs for the first time, choose “use this as default” and click “launch”.

Choose “Import-> General -> Existing project into workspace”. Select the folder ~/duckietown.

comment

Only Import -> General -> Projects from Folder or Archive, selecting ~/duckuments worked for me. JT

comment

This is not about the duckuments, it’s for duckietown - AC

If it asks about interpreters, select “auto config”.

When it shows “uncheck settings that should not be changed”, just click OK.

Slack

✎🔗

Disabling Slack email notifications

✎🔗

Most importantly, please take the time to disable email notification for Slack.

The point of Slack is that you don’t get email. You can work on Duckietown only when you have the time to do so.

to write how

Disabling Slack pop-up notification on the desktop

✎🔗

Also remove pop up notifications from the app. It should be a discrete notification that says “hey, when you have some time, look at Twist”, not “HEY HEY HEY PAY ATTENTION TO ME NOW”.

to write procedure

Because of mathjax bug

Byobu

✎🔗

Andrea

You need to learn to use Byobu. It will save you much time later.

(Alternatives such as GNU Screen are fine as well.)

Source code control with Git

✎🔗

Andrea

Setting up global configurations for Git

✎🔗

This should be done twice, once on the laptop, and later, on the robot.

These options tell Git who you are:

$ git config --global user.email "email"
$ git config --global user.name  "full name"

Also do this, and it doesn’t matter if you don’t know what it is:

$ git config --global push.default simple

git

✎🔗

to write

hub

✎🔗

Git LFS

✎🔗

This describes Git LFS.

Generic installation instructions

✎🔗

See instructions at:

https://git-lfs.github.com/

Troubleshooting

✎🔗

The binary files are not downloaded. In their place, there are short “pointer” files.

If you have installed LFS after pulling the repository and you see only the pointer files, do:

$ git lfs pull --all
Because of mathjax bug

Setup Github access

✎🔗

Andrea

This chapter describes how to create a Github account and setup SSH on the robot and on the laptop.

Add a public key to Github

✎🔗

You will do this procedure twice: once for the public key created on the laptop, and later with the public key created on the robot.

A public/private keypair already created and configured. This procedure is explained in Section 19.5 - Creating an SSH keypair.

You can access Github using the key provided.

comment

Since I am not as familiar with Linux an VIM it would have been great to say how we can get access to the public key: sudo vim /home/username/.ssh/username@host name.pub and than copy it and add it on github SL

Go to settings (Figure 29.2).

Add the public key that you created:

To check that all of this works, use the command

$ ssh -T git@github.com

The command tries to connect to Github using the private keys that you specified. This is the expected output:

Warning: Permanently added the RSA host key for IP address 'ip address' to the list of known hosts.
Hi username! You've successfully authenticated, but GitHub does not provide shell access.

If you don’t see the greeting, stop.

Repeat what you just did for the Duckiebot on the laptop as well, making sure to change the name of the file containing the private key.

Because of mathjax bug

ROS installation and reference

✎🔗

Liam

Install ROS

✎🔗

This part installs ROS. You will run this twice, once on the laptop, once on the robot.

The first commands are copied from this page.

Tell Ubuntu where to find ROS:

$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'

Tell Ubuntu that you trust the ROS people (they are nice folks):

$ sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116

Fetch the ROS repo:

$ sudo apt update

Now install the mega-package ros-kinetic-desktop-full.

$ sudo apt install ros-kinetic-desktop-full

There’s more to install:

$ sudo apt install ros-kinetic-{tf-conversions,cv-bridge,image-transport,camera-info-manager,theora-image-transport,joy,image-proc,compressed-image-transport,phidgets-drivers,imu-complementary-filter,imu-filter-madgwick}

Do not install packages by the name of ros-X, only those by the name of ros-kinetic-X. The packages ros-X are from another version of ROS.

XXX: not done in aug20 image:

Initialize ROS:

$ sudo rosdep init
$ rosdep update

rqt_console

✎🔗

to write

roslaunch

✎🔗

to write

rviz

✎🔗

to write

rostopic

✎🔗

to write

catkin_make

✎🔗

to write

rosrun

✎🔗

to write

rostest

✎🔗

to write

rospack

✎🔗

to write

rosparam

✎🔗

to write

rosdep

✎🔗

to write

roswtf

✎🔗

to write

rosbag

✎🔗

A bag is a file format in ROS for storing ROS message data. Bags, so named because of their .bag extension, have an important role in ROS. Bags are typically created by a tool like rosbag, which subscribe to one or more ROS topics, and store the serialized message data in a file as it is received. These bag files can also be played back in ROS to the same topics they were recorded from, or even remapped to new topics.

roscore

✎🔗

to write

Troubleshooting ROS

✎🔗

computer is not in your SSH known_hosts file

See this thread. Remove the known_hosts file and make sure you have followed the instructions in Section 19.3 - Local configuration.

How to install PyTorch on the Duckiebot

✎🔗

PyTorch is a Python deep learning library that’s currently gaining a lot of traction, because it’s a lot easier to debug and prototype (compared to TensorFlow / Theano).

To install PyTorch on the Duckietbot you have to compile it from source, because there is no pro-compiled binary for ARMv7 / ARMhf available. This guide will walk you through the required steps.

How to install Caffe and Tensorflow on the Duckiebot

✎🔗

Caffe and TensorFlow are popular deep learning libraries, and are supported by the Intel Neural Computing Stick (NCS).

Movidius Neural Compute Stick Install

✎🔗

How To Use Neural Compute Stick

✎🔗

Software development guide

✎🔗

This part is about how to develop software for the Duckiebot.

Because of mathjax bug

Python

✎🔗

Context managers

✎🔗

to write

Working with YAML files

✎🔗

YAML files are useful for writing configuration files, and are used a lot in Duckietown.

Duckietown code conventions

✎🔗

Python

✎🔗

Tabs

✎🔗

Never use tabs in Python files.

The tab characters are evil in Python code. Please be very careful in changing them.

Do not use a tool to do it (e.g. “Convert tabs to spaces”); it will get it wrong.

checked by what-the-duck.

Configuration

✎🔗

This chapter explains what are the assumptions about the configuration.

While the “Setup” parts are “imperative” (do this, do that); this is the “declarative” part, which explains what are the properties of a correct configuration (but it does not explain how to get there).

The tool what-the-duck (Subsection 8.1.3 - The what-the-duck program) checks some of these conditions. If you make a change from the existing conditions, make sure that it gets implemented in what-the-duck by filing an issue.

Environment variables (updated Sept 12)

✎🔗

You need to have set up the variables in Table 4.1.

The way to set these up is to add them in the file ~/.bashrc (export var="value"). Do not modify the environment.sh script.

Environment variables used by the software
variable reasonable value contains
DUCKIETOWN_ROOT ~/duckietown Software repository
DUCKIEFLEET_ROOT ~/duckiefleet Where to look for class-specific information (people DB, robots DB).
DUCKIETOWN_DATA ~/duckietown-data The place where to look for logs.
DUCKIETOWN_TMP If set, directory to use for temporary files. If not set, we use the default ( /tmp).
DUCKIETOWN_CONFIG_SEQUENCE defaults:baseline:vehicle:user The configuration sequence for EasyNode

Duckietown root directory DUCKIETOWN_ROOT

✎🔗

to write

Duckiefleet directory DUCKIEFLEET_ROOT

✎🔗

For Fall 2017, this is the the repository duckiefleet.

For self-guided learners, this is an arbitrary repository to create.

The “scuderia” (vehicle database)

✎🔗

The system needs to know certain details about the robots, such as their host names, the name of the users, etc.

This data is contained in the ${DUCKIEFLEET_ROOT}/robots/{your_branch} directory, in files with the pattern robot name.robot.yaml.

The file must contain YAML entries of the type:

owner: ID of owner
username: username on the machine
hostname: host name
description: generic description
log:
    date: comment
    date: comment

A minimal example is in Listing 4.6.

owner: censi
hostname: emma
username: andrea
description: Andrea's car.
log:
    2017-08-01: >
        Switched RPI2 with RPI3.
    2017-08-20: >
        There is something wrong with the PWM hat and the LEDs.
Minimal scuderia file emma.robot.yaml

Explanations of the fields:

  • hostname: the host name. This is normally the same as the robot name.
  • username: the name of the Linux user on the robot, from which to run programs.
  • owner: the owner’s globally-unique Duckietown ID.

The machines file

✎🔗

Make sure you already set up ROS (Section 11.3 - Set up the ROS environment on the Duckiebot).

Activate ROS:

$ cd ~/duckietown
$ source environment.sh

The machines file is created from the scuderia data using this command:

$ rosrun duckieteam create-machines

People database

✎🔗

Andrea

Describe the people database.

Modes of operation

✎🔗

There are 3 modes of operation:

  1. MODE-normal: Everything runs on the robot.
  2. MODE-offload: Drivers run on the robot, but heavy computation runs on the laptop.
  3. MODE-bag: The data is provided from a bag file, and computation runs on the laptop.
Operation modes
mode name who is the ROS master where data comes from where heavy computation happen
MODE-normal duckiebot Drivers on Duckiebot duckiebot
MODE-offload duckiebot Drivers on Duckiebot laptop
MODE-bag laptop Bag file laptop
Because of mathjax bug

Node configuration mechanisms

✎🔗

Where the config files are, how they are used.

Because of mathjax bug

Minimal ROS node - pkg_name

✎🔗

Andrea

This document outline the process of writing a ROS package and nodes in Python.

To follow along, it is recommend that you duplicate the pkg_name folder and edit the content of the files to make your own package.

Launch File

✎🔗

You should always write a launch file to launch a node. It also serves as a documentation on the I/O of the node.

Let’s take a look at launch/test.launch.

<launch>
    <node name="talker" pkg="pkg_name" type="talker.py" output="screen">

        <param name="~pub_timestep" value="0.5"/>

        <remap from="~topic_b" to="~topic_a"/>
    </node>
</launch>

For the <node>, the name specify the name of the node, which overwrites rospy.init_node() in the __main__ of talker.py. The pkg and type specify the package and the script of the node, in this case it’s talker.py.

Don’t forget the .py in the end (and remember to make the file executable through chmod).

The output="screen" direct all the rospy.loginfo to the screen, without this you won’t see any printouts (useful when you want to suppress a node that’s too talkative.)

The <param> can be used to set the parameters. Here we set the ~pub_timestep to 0.5. Note that in this case this sets the value of talker/pub_timestep to 0.5.

The <remap> is used to remap the topic names. In this case we are replacing ~topic_b with ~topic_a so that the subscriber of the node actually listens to its own publisher. Replace the line with

<remap from="~topic_b" to="talker/topic_a"/>

will have the same effect. This is redundant in this case but very useful when you want to subscribe to a topic published by another node.

Makefile system

✎🔗

Andrea

We use Makefiles to describe frequently-used commands.

Jupyter

✎🔗

Installation

✎🔗

Pull the branch 1710-place-recognition

$ cd duckietown
$ git pull

Source environment:

$ source environment.sh

Remove previous installations:

$ sudo apt remove ipython ipython-notebook

Then run:

$ pip install --user jupyter IPython==5.0

Check the versions are correct:

$ which ipython

/home/andrea/.local/bin/ipython

Check the version is correct:

$ ipython --version

5.0.0

ROS package verification

✎🔗

Andrea

This chapter describes formally what makes a conforming ROS package in the Duckietown software architecture.

Duckietown utility library

✎🔗

Bug squashing guide

✎🔗

This unit describes how to debug your programs.

Do read this accurately top-to-bottom. If you think this is too long and too verbose to read and you are in a hurry anyway: that is probably the attitude that introduced the bug.

The basic truths of bug squashing

✎🔗

What could it be?

✎🔗

How to find the bug by yourself

✎🔗

Step 0: Is it late? Go to bed.

✎🔗

If it is later than 10pm, just go to bed, and look at it tomorrow.

After 10pm, bugs are introduced, rather than removed.

Step 1: Are you in a hurry? Do it another time.

✎🔗

Bug squashing requires a clear mind.

If you are in a hurry, it’s better you do this another time; otherwise, you will not find the bug and you will only grow more frustrated.

Step 2: Make sure your environment is sane using what-the-duck

✎🔗

Finding a bug is a process of elimination of causes one-by-one, until you find the real culprit. Most of the problems come ultimately from the fact that your environment is not set up correctly.

We have a diagnostics program called what-the-duck that checks many things about the environment.

Subsection 8.1.3 - The what-the-duck program

So, first of all, run what-the-duck. Then, fix the errors that what-the-duck shows you.

This is the proper way to run what-the-duck:

$ cd ~/duckietown
$ source environment.sh
$ git checkout master
$ git pull
$ ./dependencies_for_duckiebot.sh # if you are on a Duckiebot
$ ./dependencies_for_laptop.sh  # if you are on a laptop
$ ./what-the-duck

you have to do all the steps in the precise order.

The tool also produces an HTML report about your system which you should attach to any request for help.

How to give help

✎🔗

Creating unit tests with ROS

✎🔗
catkin_make -C catkin_ws/ --pkg easy_logs
Because of mathjax bug

Continuous integration

✎🔗

These are the conventions for the Duckietown repositories.

Road Release Process

✎🔗

You have implemented a new feature or improved an existing feature in your branch devel-project_name

A robot that is configured and able to follow lanes in Duckietown according to instructions in Unit M-24 - Checkoff: Navigation

Your branch gets merged into master and doesn’t break anything

This page is about what to do once you have developed and tested your new or improved feature and you want to contribute it back to the master branch.

If your branch is not merged into master and passes all the tests it will be lost forever.

Duckietown system

✎🔗

This part describes the Duckietown algorithms and system architecture.

We do not go in the software details. The implementation details have been already talked about at length in Part K - Software development guide.

We do give links to the ROS packages implementing the functionality.

Because of mathjax bug

Teleoperation

✎🔗

add video here

Parallel autonomy

✎🔗

to write

Because of mathjax bug

Lane control

✎🔗

video here

Indefinite navigation

✎🔗

add video here

Planning

✎🔗

add video here

Coordination

✎🔗

add video here

Duckietown ROS Guidelines

✎🔗

Fall 2017

✎🔗

Welcome to the Fall 2017 Duckietown experience.

Because of mathjax bug

The Fall 2017 Duckietown experience

✎🔗

This is the first time that a class is taught jointly across 3 continents!

There are 4 universities involved in the joint teaching for the term:

  • ETH Zürich (ETHZ), with instructors Emilio Frazzoli, Andrea Censi, Jacopo Tani.
  • University of Montreal (UdeM), with instructor Liam Paull.
  • TTI-Chicago (TTIC), with instructor Matthew Walter.
  • National Chiao Tung University (NCTU), with instructor Nick Wang.

This part of the Duckiebook describes all the information that is needed by the students of the four institutions.

At ETHZ, UdeM, TTIC, the class will be more-or-less synchronized. The materials are the same; there is some slight variation in the ordering.

Moreover, there will be some common groups for the projects.

The NCTU class is undergraduate level. Students will learn slightly simplified materials. They will not collaborate directly with the other classes.

First Steps in Duckietown

✎🔗

Onboarding Procedure

✎🔗

Welcome aboard! We are so happy you are joining us at Duckietown!

This is your onboarding procedure. Please read all the steps and then complete all the steps.

If you do not follow the steps in order, you will suffer from unnecessary confusion.

Github sign up

✎🔗

If you don’t already have a Github account, sign up now.

Github signup page

Please use your full name when it asks you. Ideally, the username should be something like FirstLast or something that resembles your name.

When you sign up, use your university email. This allows to claim an educational discount that will be useful later.

Logistics for Zürich branch

✎🔗

This section describes information specific to Zürich.

✎

These are the local TAs:

  • Shiying Li (shili@student.ethz.ch)
  • Ercan Selçuk (ercans@student.ethz.ch)
  • Miguel de la Iglesia Valls (dmiguel@student.ethz.ch)
  • Harshit Khurana (hkhurana@student.ethz.ch)
  • Dzenan Lapandic (ldzenan@student.ethz.ch)
  • Marco Erni (merni@ethz.ch)

Please contact them on Slack, rather than email.

Also feel free to contact the TAs in Montreal and Chicago.

Logistics for Montréal branch

✎🔗

This unit contains all necessary info specific for students at Univeristé de Montréal.

Logistics for Chicago branch

✎🔗

Matt

This section describes information specific to TTIC and UChicago students.

✎

The course website provides a copy of the syllabus, grading information, and details on learning objectives.

✎

Classes take place on Mondays and Wednesdays from 9am-11am in TTIC Room 530. In practice, each class will be divided into an initial lecture period (approximately one hour), followed by a lab session.

The class schedule is maintained as part of the TTIC Class Diary, which includes details on lecture topics, links to slides, etc.

✎

The following is taken from the course syllabus:

The class will assess your grasp of the material through a combination of problem sets, exams, and a final project. The contribution of each to your overall grade is as follows:

  • 20%: Problem sets
  • 10%: Checkoffs
  • 20%: Participation
  • 50%: Final project (includes report and presentation). The projects will be group-based, but we will assess the contribution of each student individually.

See the course syllabus for more information on how the participation and final project grades are determined.

✎

The following is taken from the course syllabus:

Late problem sets will be penalized 10% for each day that they are late. Those submitted more than three days beyond their due date will receive no credit.

Each student has a budget of three days that they can use to avoid late penalties. It is up to the student to decide when/how they use these days (i.e., all at once or individually). Students must identify whether and how many days they use when they submit an assignment.

It is not acceptable to use code or solutions from outside class (including those found online), unless the resources are specifically suggested as part of the problem set.

You are encouraged to collaborate through study groups and to discuss problem sets and the project in person and over Slack. However, you must acknowledge who you worked with on each problem set. You must write up and implement your own solutions and are not allowed to duplicate efforts. The correct approach is to discuss solution strategies, credit your collaborator, and write your solutions individually. Solutions that are too similar will be penalized.

✎

Duckietown labs will take place at TTIC in the robotics lab on the 4th floor.

Note: TTIC and U. Chicago students in Matthew Walter’s research group use the lab as their exclusive research and office space. It also houses several robots and hardware to support them. Please respect the space when you use it: try not to distract lab members while they are working and please don’t touch the robots, sensors, or tools.

✎

Duckietown is a collaborative effort involving close interaction among students, TAs, mentors, and faculty across several institutions. The local learning assistants (LAs) at TTIC are:

  • Andrea F. Daniele (afdaniele@ttic.edu)
  • Falcon Dai (dai@ttic.edu)
  • Jon Michaux (jmichaux@ttic.edu)
Because of mathjax bug

Logistics for NCTU branch

✎🔗

Nick and Eric (Nick’s student)

This section describes information specific to NCTU students.

✎

The Duckietown Taiwan Branch Website provides some details about Duckietown Branch in NCTU-Taiwan and results of previous class in NCTU.

✎

Classes take place on Thursday from 1:20pm~4:20om in NCTU Engineering Building 5 Room 635. Each class will be devided into two sessoins. In the first session, Professor Wang will give lessons on foundamental theory and inspire students to come up more creatitive but useful ideas on final projects. In the second sessoin, TAs will give pratical lab on how to use Duckietown platform as their project platform and use ROS as their middleware toward a fantastic work.

The class schedule is maintained as part of the NCTU Class Diary, which includes details on lecture topics, links to slides, etc.

✎

The following is taken from the course syllabus:

This course aims at developing software projects usable in real-world, and focuses on “learning by doing,” “team work,” and “research/startup oriented.”. The contribution of each to your overall grade is as follows:

  • Class Participation, In Class Quiz, Problem Sets (10%)
  • Midterm Presentation (30%)
  • Final Presentation (30%)
  • Project Report and Demo Video (30%)

See the course syllabus for more information on course object and grading policy.

✎

The following is taken from the course syllabus:

Late problem sets will be penalized 10% for each day that they are late. Those submitted more than three days beyond their due date will receive no credit.

Each student has a budget of three days that they can use to avoid late penalties. It is up to the student to decide when/how they use these days (i.e., all at once or individually). Students must identify whether and how many days they use when they submit an assignment.

It is not acceptable to use code or solutions from outside class (including those found online), unless the resources are specifically suggested as part of the problem set.

You are encouraged to collaborate through study groups and to discuss problem sets and the project in person and over Slack. However, you must acknowledge who you worked with on each problem set. You must write up and implement your own solutions and are not allowed to duplicate efforts. The correct approach is to discuss solution strategies, credit your collaborator, and write your solutions individually. Solutions that are too similar will be penalized.

✎

Duckietown labs will take place at NCTU in the same place with the lecture.

Note: The course focus on “learning by doing” which means that the lab session of each class is expectially important.

✎

Duckietown is a collaborative effort involving close interaction among students, TAs, mentors, and faculty across several institutions. The local learning assistants (LAs) at NCTU are:

  • Yu-Chieh ‘Tony’ Hsiao (tonycar12002@gmail.com)
  • Pin-Wei ‘David’ Chen (ccpwearth@gmail.com)
  • Chen-Lung ‘Eric’ Lu (eric565648@gmail.com)
  • Yung-Shan ‘Michael’ Su (michael1994822@gmail.com)
  • Chen-Hao ‘Peter’ Hung (losttime1001@gmail.com)
  • Hong-Ming ‘Peter’ Huang (peterx7803@gmail.com)
  • Tzu-Kuan ‘Brian’ Chuang (fire594594594@gmail.com)
Because of mathjax bug

Git usage guide for Fall 2017

✎🔗

Repositories

✎🔗

These are the repositories we use.

Git policy for homeworks (TTIC/UDEM)

✎🔗

This does not apply to Zurich.

Homeworks will require you to write and submit coding exercises. They will be submitted using git. Since we have a university plagiarism policy (UdeM’s, TTIC/UChicago) we have to protect students work before the deadline of the homeworks. For this reason we will follow these steps for homework submission:

  1. Go here and file a request at the bottom “Request a Discount” then enter your institution email and other info.
  2. Go to exercises-fall2017
  3. Click “Fork” button in the top right
  4. Choose your account if there are multiple options
  5. Click on the Settings tab of your repostory, not your account
  6. Under “Collaborators and Teams” in the left, click the “X” in the right for the section for “Fall 2017 Vehicle Autonomy Engineers in training”. You will get a popup asking you to confirm. Confirm.

If you have not yet cloned the duckietown repo do it now:

$ git clone git@github.com:duckietown/exercises-fall2017.git

Now you need to point the remote of your exercises-fall2017 to your new local private repo. To do, from inside your already previously cloned exercises-fall2017 repo do:

$ git remote set-url origin git@github.com:GIT_USERNAME/exercises-fall2017.git

Let’s also add an upstream remote that points back to the original duckietown repo:

$ git remote add upstream git@github.com:duckietown/exercises-fall2017.git

If you type

$ git remote -v

You should now see:

origin  git@github.com:GIT_USERNAME/exercises-fall2017.git (fetch)
origin  git@github.com:GIT_USERNAME/exercises-fall2017.git (push)
upstream  git@github.com:duckietown/exercises-fall2017.git (fetch)
upstream  git@github.com:duckietown/exercises-fall2017.git (push)

Now the next time you push (without specifying a remote) you will push to your local private repo.

Git policy for project development

✎🔗

Different than the homeworks, development for the projects will take place in the Software repo since plagiarism is not an issue here. The process is:

  1. Create a branch from master

  2. Develop code in that branch (note you may want to branch your branches. A good idea would be to have your own “master”, e.g. “your_project-master” and then do pull requests/merges into that branch as things start to work)

  3. At the end of the project submit a pull request to master to merge your code. It may or may not get merged depending on many factors.

Because of mathjax bug

Organization

✎🔗

The following roster shows the teaching staff.

Andrea Censi
Liam Paull
Jacopo Tani
First Last
Emilio Frazzoli
First Last
First Last
First Last
First Last
First Last
First Last
First Last
First Last
First Last
First Last
Greta Paull
Kirsten Bowser

Staff: To add yourself to the roster, or to change your picture, add a YAML file and a jpg file to the duckiefleet-fall2017 repository. in the people/staff directory.

The Activity Tracker

✎🔗

Link to Activity Tracker

The sheet called “Activity Tracker” describes specific tasks that you must do in a certain sequence. Tasks include things like “assemble your robot” or “sign up on Github”.

The difference between the Areas sheet and the Task sheet is that the Task sheet contains tasks that you have to do once; instead, the Areas sheet contains ongoing activities.

In this sheet, each task is a row, and each person is a column. There is one column for each person in the class, including instructors, TAs, mentors, and students.

You have two options:

  • Only use the sheet as a reference;
  • Use the sheet actively to track your progress. To do this, send a message to Kirsten with your gmail address, and add yourself.

Each task in the first column is linked to the documentation that describes how to perform the task.

The colored boxes have the following meaning:

  • Grey: not ready. This means the task is not ready for you to start yet.
  • Red: not started. The person has not started the task.
  • Blue: in progress. The person is doing the task.
  • Yellow: blocked. The person is blocked.
  • Green: done. The person is done with the task.
  • n/a: the task is not applicable to the person. (Certain tasks are staff-only.)

The Areas sheet

✎🔗

Please familiarize yourself with this spreadsheet and bookmark it in your browser.

The sheet called “Areas” describes the points of contact for each part of this experience. These are the people that can offer support. In particular, note that we list two points of contact: one for America, and one for Europe. Moreover, there is a link to a Slack channel, which is the place where to ask for help. (We’ll get you started on Slack in just a minute.)

Because of mathjax bug

Getting and giving help

✎🔗

Slack Channels

✎🔗

This page describes all of the helpful Slack channels and their purposes so that you can figure out where to get help.

Channels

✎🔗

You can also easily join the ones that you are interested in by clicking the links in this message.

Duckietown Slack Channels
Channel Purpose
help-accounts Info about necessary accounts, such as Slack, Github, etc.
help-assembly Help putting your robot together
help-camera-calib Help doing the intrinsic and extrinsic calibration of your camera
help-duckuments Help compiling the online documentation
help-git Help with git
help-infrastructure Help with software infrastructure, such as Makefiles, unit tests, continuous integration, etc.
help-laptops Help getting your laptop setup with Ubuntu 16.04
help-parts Help getting the parts for the robot or replacement parts if you broke something
help-robot-setup Help getting the robot setup to do basic things like be driven with a joystick
help-ros Help with the Robot Operating System (ROS)
help-wheel-calib Help doing your odometry calibration
comment

Note that we can link directly to the channels. (See list in the org sheet.) -AC

Because of mathjax bug

Zürich branch diary

✎🔗

Wed Sep 20: Welcome to Duckietown!

✎🔗

This was an introduction meeting.

Monday Sep 25: Introduction to autonomy

✎🔗

Monday Sep 25, late at night: Onboarding instructions

✎🔗

At some late hour of the night, we sent out the onboarding instructions.

Section 2.1 - Onboarding Procedure

Please complete the onboarding questionnaire by Tuesday, September 26, 15:00.

Wednesday Sep 27: Duckiebox distribution, and getting to know each other

✎🔗

Today we distribute the Duckieboxes and we name the robots. In other words, we perform the Duckiebox ceremony.

  • getting to know each other;
  • naming the robots;
  • distribute the Duckieboxes.

If you cannot make it to this class for the Duckiebox distribution, please inform the TA, to schedule a different time.

Preparation, step 1: choose a name for your robot

✎🔗

Before arriving to class, you must think of a name for your robot.

Here are the constraints:

  • The name must work as a hostname. It needs to start with a letter, contains only letters and numbers, and no spaces or punctuation.
  • It should be short, easy to type. (You’ll type it a lot.)
  • It cannot be your own name.
  • It cannot be a generic name like “robot”, “duckiebot”, “car”. It cannot contain brand names.

Preparation, step 2: prepare a brief elevator pitch

✎🔗

As members of the same community, it is important to get to know a little about each other, so to know who to rely on in times of need.

During the Duckiebox distribution ceremony, you will be asked to walk up to the board, write your name on it, and introduce yourself. Keep it very brief (30 seconds), and tell us:

  • what is your professional background and expertise / favorite subject;
  • what is the name of your robot;
  • why did you choose to name your robot in that way.

You will then receive a Duckiebox from our senior staff, a simple gesture but of sempiternal glory, for which you have now become a member of the Duckietown community. This important moment will be remembered through a photograph. (If in the future you become a famous roboticist, we want to claim it’s all our merit.)

Finally, you will bring the Duckiebox to our junior staff, who will apply labels with your name and the name of the robot. They will also give you labels with your robot name for future application on your Duckiebot.

Thursday Sep 28: Misc announcements

✎🔗

We created the channel #ethz-chitchat for questions and other random things, so that we can leave this channel #ethz-announcements only for announcements.

We sent the final list to the Department; so hopefully in a couple of days the situation on MyStudies is regularized.

The “lab” time on Friday consists in an empty room for you to use as you wish, for example to assembe the robots together. In particular, it’s on the same floor of the Duckietown room and the IDSC lab.

The instructions for assembling the Duckiebots are here. Note that you don’t have to do the parts that we did for you: buying the parts, soldering the boards, and reproducing the image.

Expected progress: We are happy if we see everybody reaching up to Unit C-14 - RC+camera remotely by Monday October 9. You are encouraged to start very early; it’s likely that you will not receive much help on Sunday October 8…

Because of mathjax bug

Sep 28: some announcements

✎🔗

A couple of announcements:

  1. We created #ethz-chitchat for questions and other random things, so that we can leave this channel #ethz-announcements only for announcements.

  2. MyStudies should be updated with everybody’s names.

  3. The “lab” time tomorrow consists in an empty room for you to use as you wish, for example to assemble the robots together. In particular, it’s on the same floor of the Duckietown room and the IDSC lab.

  4. The instructions for assembling the Duckiebots are here. Note that we did for you step I-2 (buying the parts) and I-3 (soldering the boards); and I-6 is optional.

  5. We are happy if we see everybody reaching I-13 by the Monday after next. I encourage you to start sooner than later.

  6. I see only 30 people in this channel instead of 42. Please tell your friends that now all the action is on Slack.

Oct 01 (Mon): Announcement

✎🔗

It looks like that the current documentation is misleading in a couple of points. This is partially due to the fact that there is some divergence between Chicago, Montreal and Zurich regarding (1) the parts given out and (2) the setup environment (which networks are available). We did the simple changes (e.g. removing the infamous part 6), but we need some more time to review the other issues. At this point, the best course of action is that you enjoy your weekend without working on Duckietown, while we spend the weekend fixing the documentation.

Oct 02 (Mon): Networking, logical/physical architectures

✎🔗

Oct 04 (Wed): Modeling

✎🔗

Oct 09 (Mon): Autonomy architectures and version control

✎🔗

Oct 11 (Wed): Computer vision and odometry calibration

✎🔗

Oct 13 (Fri): new series of tasks out

✎🔗

Taking a video of the joystick control

✎🔗

Please take a video of the robot as it drives with joystick control, as described in Section 17.7 - Upload your video and upload it according to the instructions.

Example of a great video, but with a nonconforming Duckiebot.

Camera calibration

✎🔗

Go forth and calibrate the camera! And get help in #help-camera-calib.

Wheel calibration

✎🔗

This is not ready yet! will be ready in a day or so.

Taking a log check off

✎🔗

Follow the instructions here to learn how to take a log.

Data processing exercises

✎🔗

See the list of exercises here.

Get help in #ex-data-processing.

Current deadlines for Zurich
Robot assembly overdue
Robot/laptop configuration overdue
Subsection 11.13.1 - Taking a video of the joystick control Monday Oct 16
Subsection 11.13.2 - Camera calibration Friday Oct 20
Subsection 11.13.3 - Wheel calibration not ready yet
Subsection 11.13.4 - Taking a log check off Wed Oct 18
Subsection 11.13.5 - Data processing exercises Monday Oct 23

Oct 16 (Mon): Line detection

✎🔗

Oct 18 (Wed): Feature extraction

✎🔗

Oct 20 (Fri): Lab session

✎🔗

Oct 23 (Mon) Filtering I

✎🔗

Oct 25 (Wed) Filtering II

✎🔗

a - Lectures (Particle Filter) PowerPoint presentation, PDF.

b - Lectures (Lane Filter) PowerPoint presentation, PDF.

Nov 1 (Wed) Control Systems

✎🔗

a - Lectures (Control Systems Module I) PowerPoint presentation, PDF.

b - Lectures (Control Systems Module II) PowerPoint presentation, PDF.

Points to be noted - Running what-the-duck on laptop and Duckiebot is mandatory. It helps save time in debugging errors and also is a standard way to ask for help from the staff. Keep repeating it periodically so as to keep the data up-to date - For the people lacking calibrated wheels, this serves as a reminder to calibrate the wheels and keep their duckiebot up-to date - It is advised to fill the lecture feedback form (Feedback form), so as to increase the effectiveness of the lectures - Always check the consistency of the camera calibration checkerboard before camera calibration (one has to check for the correct square size and correct distance between world and checkerboard reference)

Nov 6 (Mon) Project Pitches

✎🔗

Lecture Project Pitches PDF.

Nov 8 (Wed) Motion Planing

✎🔗

Lecture Motion Planing PDF.

A few references for planning of Andrea Censi:

Nov 13 (Mon) Project Team Assignments

✎🔗
  • First Lecture: Project Team Assignments PDF.
  • Second Lecture: First meeting of the Controllers group –> Filling out the Preliminary Design Document

Nov 15 (Wed) Putting things together

✎🔗
  • First Lecture: Putting things together PDF.
  • Second Lecture: Second meeting of the Controllers group –> Filling out the Preliminary Design Document

Nov 22 (Wed) Fleet Control

✎🔗
  • Lecture: Fleet Control in Autonomous Mobility on Demand PDF.

Nov 27 (Mon) Inermediate design Report

✎🔗

  • First Lecture: The intermediate Design report is introduced heretemplate-int-report. It is due on Monday 4th of December.

  • Second Lecture was left for project discussion and interaction.

Nov 29 (Wed) Fleet Control

✎🔗

  • First Lecture: Clausdio finished Fleet Control in Autonomous Mobility on Demand PDF.

  • Second Lecture Julian Presented the state of the art in data driven vs Model driven robotics. PDF

Because of mathjax bug

Montréal branch diary

✎🔗

Oct 30 (Mon)

✎🔗

XXX

Nov 01 (Wed)

✎🔗

XXX

Nov 06 (Mon)

✎🔗

XXX

Nov 08 (Wed)

✎🔗

XXX

Nov 13 (Mon)

✎🔗

XXX

Nov 15 (Wed)

✎🔗

XXX

Nov 20 (Mon)

✎🔗

XXX

Nov 22 (Wed)

✎🔗

XXX

Nov 27 (Mon)

✎🔗

XXX

Nov 29 (Wed)

✎🔗

XXX

Dec 04 (Mon)

✎🔗

XXX

Dec 06 (Wed)

✎🔗

XXX

Dec 11 (Mon)

✎🔗

XXX

Chicago branch diary

✎🔗

Classes take place on Mondays and Wednesdays from 9am-11am in TTIC Room 530.

NCTU branch diary

✎🔗

Classes take place on Thursday from 1:20pm-4:20pm in NCTU Engineering Building 5 Room 635.

Slack Channels

✎🔗

This page describes all of the helpful Slack channels and their purposes so that you can figure out where to get help.

Guide for TAs

✎🔗

Checkoff: Assembly and Configuration

✎🔗

The first job is to get your Duckiebot put together and up and running.

Upload your video

✎🔗

You should record a video demonstrating that your Duckiebot is up and running. Brownie points for creative videos. Please upload your videos via the following URL:

Chicago: upload your video

Zurich: upload your video

Because of mathjax bug

Checkoff: Take a Log

✎🔗

Unit M-17 - Checkoff: Assembly and Configuration

A verified log in rosbag format uploaded to Dropbox.

#help-logging

Montreal deadline: Oct 4, 11pm

Zurich deadline: Oct 20, 17:00

Checkoff: Robot Calibration

✎🔗

Unit M-17 - Checkoff: Assembly and Configuration

That you have correctly cloned and followed the git procedure outline in Unit M-7 - Git usage guide for Fall 2017

That you have correctly setup your environment variables according to Section 4.1 - Environment variables (updated Sept 12)

You robot calibrations (wheels and camera (x2)) are merged to git through a PR

Slack channels: #help-wheel-calib, #help-camera-calib

Homework: Data Processing (UdeM)

✎🔗

Unit M-18 - Checkoff: Take a Log

Unit J-30 - ROS installation and reference

Ability to perform basic operations on images

Build your first ROS package and node

Ability to process imagery live

#ex-data_processing

Montreal deadline: Oct 4, 11:00pm

Homework: Data Processing (TTIC)

✎🔗

Unit J-30 - ROS installation and reference

Ability to perform basic operations on images

Build your first ROS package and node

Ability to process imagery live

#help-data-processing

TTIC deadline: Friday, October 13 11:59pm CT

Exercises: Data Processing (Zurich)

✎🔗

Unit J-30 - ROS installation and reference

Ability to perform basic operations on images

Build your first ROS package and node

Ability to process imagery live

Slack channel to get help: #ex-data-processing

Homework: Augmented Reality

✎🔗

Unit J-30 - ROS installation and reference

Unit C-17 - Camera calibration

Ability to project fake things from an image back into the world

#ex-augmented_reality

Montreal deadline: Oct 27, 11:00pm
Chicago deadline: Oct 27, 11:59pm
Zurich deadline: Oct ???, 11:59pm

Checkoff: Navigation

✎🔗

Unit M-19 - Checkoff: Robot Calibration

Unit M-18 - Checkoff: Take a Log

2 logs of your robot autonomously navigating Duckietown

Montreal Deadline: Nov 15, 11pm

Chicago Deadline: Nov 15, 11pm

#help-navigation

Homework: Lane Filtering

✎🔗
Montreal deadline: Nov 17, 11:00pm
Chicago deadline: Nov 17, 11:59pm

#ex-filtering

Guide for mentors

✎🔗

Liam?

Because of mathjax bug

Project proposals

✎🔗

to write

Because of mathjax bug

System Architecture

✎🔗

Mission 1

✎🔗

Ensure that the development and integration of the projects into the system goes smoothly and that the resulting system makes sense, and is useful for future duckierations (duckie + generations).

Mission 2

✎🔗

Where there is a system, there is a want (nay, need) for optimisation. Describing a system’s performance and resource requirements in a quantifiable way is a critical part of being able to benchmark modules and optimise the system.

Mission 2 is to formalise the description of the system characteristics, so that eventually the system performance can be optimised for some given resources.

Template of a project

✎🔗

Make a copy of this document before editing.

The Map Description

✎🔗

The map to be used in the Fall 2017 class is shown in Figure 30.1.

The map to be used in the Fall 2017 class

The editable keynote file is in this directory of the duckuments repo. The ids on the signs correspond to the Apriltag IDs. For more details see Unit D-3 - Signage.

Because of mathjax bug

Fall 2017 projects

✎🔗

Welcome to the Fall 2017 projects.

Instructions for using the template

✎🔗

  1. Make a copy of the template 10_templates folder and paste it inside /atoms_85_fall2017_projects.

  2. Rename the folder to the next available integer followed by the short group name. E.g.: 10_templates becomes 11_first_group_name for the first group, then 12_second_group_name for the second, and so forth.

  3. Edit the 10-preliminary-design-document-template file name by substituting template with group-name

  4. Open the preliminary design document and personalize the template to your group.

All groups have got their unique ID number and folders are renamed according to the following table. You are allowed and encouraged to use short names. Please merge from the master. New pull requests conflicting to this table will be rejected.

Group name: preliminary design document

✎🔗

Group name: intermediate report

✎🔗

It’s time to commit on what you are building, and to make sure that it fits with everything else.

This consists of 3 parts:

  • Part 1: System interfaces: Does your piece fit with everything else? You will have to convince both system architect and software architect and they must sign-off on this.

  • Part 2: Demo and evaluation plan: Do you have a credible plan for evaluating what you are building? You will have to convince the VPs of Safety and they must sign-off on this.

  • Part 3: Data collection, annotation, and analysis: Do you have a credible plan for collecting, annotating and analyzing the data? You will have to convince the data czars and they must sign-off on this.

Intermediate Report Supervisors
System Architects Sonja Brits, Andrea Censi
Software Architects Breandan Considine, Liam Paull
Vice President of Safety Miguel de la Iglesia, Jacopo Tani
Data Czars Manfred Diaz, Jonathan Aresenault

Group name: final report

✎🔗

The objective of this report is to bring justice to your hard work during the semester and make so that future generations of Duckietown students may take full advantage of it. Some of the sections of this report are repetitions from the preliminary and intermediate design document (PDD, IDD respectively).

The final result

✎🔗

Let’s start from a teaser.

  • Post a video of your best results (e.g., your demo video)

Add as a caption: see the operation manual to reproduce these results.

Mission and Scope

✎🔗

Now tell your story:

Define what is your mission here.

Motivation

✎🔗

Now step back and tell us how you got to that mission.

  • What are we talking about? [Brief introduction / problem in general terms]

  • Why is it important? [Relevance]

Existing solution

✎🔗
  • Was there a baseline implementation in Duckietown which you improved upon, or did you implemented from scratch? Describe the “prior work”

Opportunity

✎🔗
  • What didn’t work out with the existing solution? Why did it need improvement?

Examples: - there wasn’t a previous implementation - the previous performance, evaluated according to some specific metrics, was not satisfactory - it was not robust / reliable - somebody told me to do so (/s)

  • How did you go about improving the existing solution / approaching the problem? [contribution]
  • We used method / algorithm xyz to fix the gap in knowledge (don’t go in the details here)
  • Make sure to reference papers you used / took inspiration from

Preliminaries (optional)

✎🔗
  • Is there some particular theorem / “mathy” thing you require your readers to know before delving in the actual problem? Add links here.

Definition of link: - could be the reference to a paper / textbook (check here how to add citations) - (bonus points) it is best if it is a link to Duckiebook chapter (in the dedicated “Preliminaries” section)

Definition of the problem

✎🔗

Up to now it was all fun and giggles. This is the most important part of your report: a crisp mathematical definition of the problem you tackled. You can use part of the preliminary design document to fill this section.

Make sure you include your: - final objective / goal - assumptions made (including contracts with “neighbors”) - quantitative performance metrics to judge the achievement of the goal

Contribution / Added functionality

✎🔗

Describe here, in technical detail, what you have done. Make sure you include: - a theoretical description of the algorithm(s) you implemented - logical architecture (refer to IDD template for description) - software architecture (refer to IDD template for description) - details on the actual implementation where relevant (how does the implementation differ from the theory?) - any infrastructure you had to develop in order to implement your algorithm - If you have collected a number of logs, add link to where you stored them

Feel free to create subsections when useful to ease the flow

Formal performance evaluation / Results

✎🔗

Be rigorous!

  • For each of the tasks you defined in you problem formulation, provide quantitative results (i.e., the evaluation of the previously introduced performance metrics)
  • Compare your results to the success targets. Explain successes or failures.
  • Compare your results to the “state of the art” / previous implementation where relevant. Explain failure / success.
  • Include an explanation / discussion of the results. Where things (as / better than / worst than) you expected? What were the biggest challenges?

Future avenues of development

✎🔗

Is there something you think still needs to be done or could be improved? List it here, and be specific!

Because of mathjax bug

The Heroes quests: preliminary report

✎🔗

The “Heroes” team is a special task force with the responsibility to make sure that “everything works” and create a smooth experience for the rest of the teams, in terms of developing own projects, integration with other teams and documentation. Apart from that, each of the heroes will also have their own individual quest…

The Heroes - System Architecture: final report

✎🔗

System Architecture refers to the high-level conceptual model that defines the structure and behaviour of a system. There are different ways to get an insight into the architecture of a system, for example functional decomposition diagrams, package composition, or a Finite State Machine diagram.

The final result

✎🔗

The System Architecture project helped to ensure that the Fall 2018 projects integrate into the existing system. The role of the System Architect was to identify and solve problems that arose during the project development and integration, as well as influencing the high-level design of the system.

In Duckietown, the Finite State Machine (FSM) diagram plays an important role in determining how the higher-level system behaves in different scenarios. The FSM defines which states the system can be in, and which functionalities must be active in which states. During the project, the need for development on the FSM arose. Below is the resulting updated Finite State Machine (FSM) diagram.

FSM diagram

The Finite State Machine

Mission and Scope

✎🔗

The System Architecture project was not a clearly defined work package, but rather a responsibility to ensure smooth development and integration of the new projects into the existing system.

Motivation

✎🔗

With many teams working on many different parts of the system, chaos is inevitable (without divine intervention). The role of the System Architect was to ensure that development and intergation of new projects goes smoothly, by addressing interface, contract and dependency issues between the teams.

During the project, the need arose for an updated version of the FSM.

Existing solution

✎🔗

Duckietown already had an existing system architecture. As metioned before, the FSM is closely related to the system architecture, since it defined the high-level behaviour of the system. There was an existing infrastructure for the FSM, which could be modified to include the newly developed functionality. The infrastructure consisted of the fsm ROS package, the configuration of the FSM in the form of .yaml files, as well as a tool to visualise the FSM structure.

✎

The fsm package consists of two nodes, namely the fsm_node and the logic_gate_node. The fsm_node is in charge of determining the current state and computing state transitions, and the logic_gate_node acts as a helper node to the fsm_node. For more information, see the README of the fsm package, found at 20-indefinite-navigation/fsm/README.md.

✎

While the fsm package handles the computation of state transitions, the FSM states and transitions can be configured using the supplied .yaml files. The fsm package then reads the configuration in order to know which states and transitions are available in the system. This allows for separation of the computation and configuration of the FSM.

✎

There exists a tool to parse and visualise the FSM configuration (in the form of an FSM diagram), found at 00-infrastructure/ros_diagram/parse_fsm.py. The tool parses the .yaml configuration file and outputs a .dot format graph, which can be converted to .png.

Opportunity

✎🔗

During Fall 2018, many new projects were being developed by multiple teams. This created the need for someone to ensure harmony between the projects and the system during the process.

Duckietown was being expanded with new functionalities such as parking and deep learning lane following. The previous FSM was not equipped to deal with the new features, therefore it had to be further developed.

Definition of the problem

✎🔗

Ensuring that development of the new projects integrate into the existing system with as little as possible chaos. This implies ensuring that all teams understand their package’s objective and their impact on the bigger system.

Contribution / Added functionality

✎🔗

The contribution of this project was in the form of both organisational activities, as well as software development. System integration of project modules. The approach can be summarised as follows:

  • Become one with the goals of Duckietown
  • Be familiar with the current system architecture
  • Track changes and identify how they influence the system.
  • Keep close communication with project teams.
  • Act as middlemand/helper to facilitate negotiation of contracts in and between groups.
  • Monitor status of projects to find possible problems.

Formal performance evaluation / Results

✎🔗

Be rigorous!

  • For each of the tasks you defined in you problem formulation, provide quantitative results (i.e., the evaluation of the previously introduced performance metrics)
  • Compare your results to the success targets. Explain successes or failures.
  • Compare your results to the “state of the art” / previous implementation where relevant. Explain failure / success.
  • Include an explanation / discussion of the results. Where things (as / better than / worst than) you expected? What were the biggest challenges?

Future avenues of development

✎🔗

The existing framework for the FSM made it relatively easy to update it to include new functionalities (once you’ve decided on the system architecture). The FSM is configured using .yaml. files, which are then loaded into the fsm_node.

Development of the updated FSM was done in response to a need before demo day, and while it has been tested on its own, it has not been tested thoroughly with all other parts of the system yet.

Because of mathjax bug

PDD - Smart City

✎🔗

PDD - System Identification

✎🔗

System Identification: Intermediate Report

✎🔗

The Controllers: preliminary report

✎🔗

The Controllers: Intermediate Report

✎🔗

Intermediate Report Supervisors
System Architects Sonja Brits, Andrea Censi
Software Architects Breandan Considine, Liam Paull
Vice President of Safety Miguel de la Iglesia, Jacopo Tani
Data Czars Manfred Diaz, Jonathan Aresenault

Conventions used in the following document
Variable Description
$d_{ref}$ Reference distance from center of lane
$d_{act}$ Actual distance from center of lane
$d_{est}$ Estimated distance from center of lane
$\theta_{act}$ Actual angle between robot and center of lane
$\theta_{est}$ Estimated angle between robot and center of lane
$c_{act}$ Actual curvature of lane
$c_{est}$ Estimated curvature of lane
$c_{ref}$ Reference curvature of the path to follow
$v_{ref}$ Reference velocity

image

The Controllers: final report

✎🔗

The final result

✎🔗

The video is at https://vimeo.com/250511342.

The Controllers Demo Video

See the operation manual to reproduce these results.

Mission and Scope

✎🔗

IMPERIUM ET POTESTAS EST
(With control comes power)

Our Mission was to make lane following more robust to model assumptions and Duckietown geometric specification violations and provide control for a different reference.

Motivation

✎🔗

In Duckietown, Duckiebots are cruising on the streets and also Duckies are sitting on the sidewalk waiting for a Duckiebot to pick them up. To ensure a baseline safety of the Duckiebots and the Duckies, we have to make sure the Duckiebots are able to follow the lane (or a path on intersections and in parking lots) and stop in front of red lines. For instance, the Duckiebot is driving on the right lane. It should never cross the centerline to avoid any collisions with an oncoming Duckiebot.

The overall goal of our project is to stay in the lane while driving and stopping in front of a red line. Due to the tight time plan, we focused on improving the existing code and benchmarking the tasks. In order to let the Duckiebot drive to a given point, the robot has to know where it is in the lane, calculate the error and define a control action to reach the target. To retrieve the location and orientation information, a pose estimator is implemented. The estimator receives line segments from the image pipeline with information about line tape colour (white, yellow, red) (Figure 11.2) and whether the segment is on the left or right edge of the line tape. Using those information, we determine if the Duckiebot is inside or outside the lane, how far it is from the middle of the lane and at what angle it stands. The relative location to the middle of the lane and the orientation of the Duckiebot are passed on to the controller. In order to minimize the error, the controller calculates the desired velocity and heading of the Duckiebot using the inputs and controller parameters. The importance of our project in the framework “Duckietown” was obvious, as it contains the fundamental functionality of autonomous driving. Furthermore, many other projects relied on our project’s functionality such as obstacle avoidance, intersection navigation or parking of a Duckiebot. We had to ensure that our part is robust and reliable.

Image with Line Segments
Image with Line Segments, $d_{err}$ and $\phi_{err}$ displayed.

Existing solution

✎🔗
Curve plot
Pose of Duckiebot in a curve element.

From last year’s project, the baseline implementation of a pose estimator and a controller were provided to us for further improvement. The prior pose estimator was designed to deliver the pose for a Duckiebot on straight lanes only. If the Duckiebot was in or before a curve and in the middle of the lane, the estimated pose showed an offset d, see definition of d in figure below. The existing controller worked reasonably on straight lines. Although, due to the inputs from the pose estimator to the controller, the Duckiebot overshot in the curves and crossed the left/right line during or after the curve.

The video is at https://vimeo.com/256663571.

Old vs. new controller

Opportunity

✎🔗

In the previous implementation, the lane following was not guaranteed on curved lane segments, because the Duckiebot often left the lane while driving in the curve or after the curve. Although the Duckiebot sometimes returned correctly to the right lane after leaving it and continued following the lane, robust lane following was not provided. On straight lanes, the Duckiebot frequently drove with a large static offset from the center of the lane. The previously implemented pose estimator and controller left room for improvement.

Further, the previous lane controller was not benchmarked for robustness nor for performance, therefore we defined various tests to benchmark the previous controller and our updated solution. During the project, we continuously tested our code with the entire lane following pipeline for best practice and compared our implemented solution to the existing one to record the improvement.

Our Scope was first of all to enable controlled autonomous driving of the Duckiebot on straight lane segments and curved lane segments which are in compliance with the geometry defined in Duckietown Appearance Specifications. Further, we wanted to enhance the robustness to arbitrary geometry of lane width or curvature of the lane to ensure the autonomous driving of the Duckiebot in an individual Duckietown setup. We also tackled the detection and stopping at red (stop) lines. With the previous implementation, the Duckiebot stopped rather at random points in front of the red line. We wanted to improve the implementation, to ensure a stop in the middle of the lane, in a predefined range and at a straight angle to the red line. As the Duckietown framework is a complex system involving various functionalities such as obstacle avoidance and intersection navigation, our lane following pipeline provides the basic function for those functionalities and it has to be able to interact with the modules of other teams. Hence, it was also our duty to design an interface which can receive and apply information from other modules. For example, our controller can take reference d from obstacle avoidance, intersection crossing and parking. For intersection navigation and parking, our controller needs additionally the pose estimation and a curvature from the navigators and the parking team respectively.

Out of scope was:

  • Pose estimation and curvature on Intersections (plus navigation / coordination)
  • Model of Duckiebot and uncertainty quantification of parameters (System Identification)
  • Object avoidance involving going to the left lane
  • Extraction and classification of edges from images (anti-instagram)
  • Any hardware design
  • Controller for Custom maneuvers (e.g. Parking, Special intersection control)
  • Robustness to non existing line

Preliminaries (optional)

✎🔗

Definition of the problem

✎🔗

Our final objective is to keep the Duckiebots at a given distance d from the center of the lane, on straight and curved roads, under bounded variations of the city geometric specifications.

The project was on the bottom line, taking the line segments which gave information about the line colour and the segment positions to estimate the Duckiebot’s pose and return a command for the motors to steer the robot to the center of the lane. After roughly analysing the existing solution, we divided the work load into two topics pose estimation and controller to enable parallel dealing with the problems in the short period of time.

In our Preliminary Design Document and Intermediate Report, we have listed all variables and their definitions, as well as all system interfaces with other groups and assumptions we made. Due to limitation of time and different priorities of other teams, some integrations with other teams are not yet activated but they are already prepared in our code (some of it commented out).

Contribution / Added functionality

✎🔗

Formal performance evaluation / Results

✎🔗

We evaluated the improvement of the performance with help of several tests. The evaluation procedure are defined in our Intermediate Report. The main benchmark feature was the average deviation from tracking reference during a run (distance to middle lane) and the standard deviation of the same value. We also benchmarked the deviation from the heading angle as well but since the bot is mainly controlled according to the deviation of the tracking distance, it was the main feature to lead our development. Benchmarking in general occurred by letting the Duckiebot run a specific experiment and recording a rosbag. We wrote a distinct offline benchmarking application mentioned above, that analyzes the rosbag containing the recorded values and creates plots with the extracted information about tracking distance and heading angle over the run.

Furthermore, we assessed the performance of the Duckiebots in the following dimensions:

  • Estimator:
    • Static lane pose estimation benchmark
    • Static curve pose estimation benchmark
    • Image resolution benchmark
    • Segment interpolation benchmark
    • Curvature estimation benchmark
  • Controller:
    • Stop at red line benchmark
    • Controller benchmark
    • Non-conforming curve benchmark

Future avenues of development

✎🔗

As there is always more to do and the performance for both the controller and the estimator can still be further enhanced we list in this section some suggestions for next steps to take.

PDD - Saviors

✎🔗

The Saviors: intermediate report

✎🔗

The Saviors: Final Report

✎🔗

This is the final report of the fall 2017 Saviors group from ETH Zurich, namely Fabio Meier (fmeier@ethz.ch), Julian Nubert (nubertj@ethz.ch), Fabrice Oehler (foehler@ethz.ch) and Niklas Funk (nfunk@ethz.ch). We enjoyed contributing to this great project and in case there are any open questions left after having read this report, do not hesitate to contact us.

The Final Result

✎🔗

The Saviors Teaser:

Vimeo

See the operation manual to reproduce these results.

The code description can be found here in the Readme.

Mission and Scope

✎🔗

“URBEM ANATUM TUTIOS FACIENDA (EST) - MAKE DUCKIETOWN A SAFER PLACE”

The goal of Duckietown is to provide a relatively simple platform to explore, tackle and solve many problems linked to autonomous driving. “Duckietown” is simple in the basics, but an infinitely expandable environment. From controlling single driving Duckiebots until complete fleet management, every scenario is possible and can be put into practice. Due to the previous classes and also the great work of many volunteers, many software packages were already developed and provided a solid basis. But something was still missing.

Motivation

✎🔗

So far, none of the mentioned modules was capable of reliably detecting obstacles and reacting to them in real time. We were not the only ones who saw a problem in the situation at that time: “Ensuring safety is a paramount concern for the citizens of Duckietown. The city has therefore commissioned the implementation of protocols for guaranteed obstacle detection and avoidance.”[31]. Therefore the foundation of our complete module lies in the disposal of this shortcoming. Finding a good solution for this safety related and very important topic helped us to stay motivated every day we were trying to improve our solution.

The goal of our module is to detect obstacles and react accordingly. Due to the limited amount of time, we focused the scope of our module to two points:

1. In terms of detection, on the one hand we focused to reliably detect yellow duckies and therefore to saving the little duckies that want to cross the road. On the other hand we had to detect orange cones to not crash into any construction site in Duckietown.

2. In terms of reacting to the detected obstacles we were mainly restricted by the constraint given by the controllers of our Duckiebots, who do not allow us to cross the middle of the road. This eliminated the need of also having to implement a Duckiebot detection algorithm. So we focused on writing software which tries to avoid obstacles within our own lane if it is possible (e.g. for avoiding cones on the side of the lane) and to stop otherwise.

Besides aforementioned restrictions and simplifications we faced the general problem of detecting obstacles given images from a monocular RGB camera mounted at the front of our Duckiebot and reacting to them properly without crashing or erroneously stopping the Duckiebot. Both processes above have to be implemented and have to run on the Raspberry Pis in real time. Due to the strong hardware limitations, we decided to not use any learning algorithms for the obstacle detection part. As it later transpired, a working “hard coded” software needs thorough analysis and understanding of the given problem. However, in the future, considering additional hardware like e.g. Tung, “Google offers Raspberry Pi owners this new AI vision kit” (2017), this decision might have to be reevaluated.

In practice a well working obstacle detection is one of the most important parts of an autonomous system to improve the reliability of the outcome even in unexpected situations. Therefore the relevance of an obstacle detection in a framework like “Duckietown” is very important. Especially because the aim of “Duckietown” is to simulate the real world as realistic as possible and also in other topics such as fleet planning, a system with obstacle detection behaves completely different than a system without.

Existing Solution

✎🔗

There was a previous implementation from the MIT classes in 2016. Of course we had a look into the old software and found out that one step of them was quite similar to ours: They based their obstacle detection on the colors of the obstacles. Therefore they also did their processing in the HSV color space as we did. Further information on why filtering colors in the HSV space is advantageous can be found in the Theory Chapter.

Nevertheless, we implemted our solution from scratch and didn’t base ours on any further concepts found in their software. That is why you won’t find any further similarites between the two implementations. The reasons for implementing our own code from scratch can be found in the next section Opprtunity. In short, last year’s solution considered the image given the original camera’s perspective and tried to classify the objects based on their contour. We are using a very different approach concerning those two crucial parts as you can see in the Contribution section.

Opportunity

✎🔗

From the beginning it was quite clear that the old software was not working reliable enough. The information we have been given was that it was far off detecting all obstacles and that there were quite a few false positives: It detected yellow line segments in the middle of the road as obstacles (color and size are quite similar to the ones of typical duckies) which led to a stopping of the car. Furthermore, extracting the contour of every potential obstacle is highly computationally expensive. As mentioned, we had a look into the software and tried to understand it as well as possible but because it was not documented at all we couldn’t go much into detail. On top of that, from the very beginning we had a completely different idea of how we wanted to tackle these challenges.

We also tried to start their software but we couldn’t make it run after a significant amount of time. The readme file didn’t contain any information and the rest of the software was not documented as well. This also reinforced us in our decision to write our own implementation from scratch.

Preliminaries

✎🔗

Since our task was to reliably detect obstacles using a monocular camera only, we mainly dealt with processing the camera image, extracting the needed information, visualizing the results and to act accordingly in the real world.

For understanding our approach we tried to explain and summarize the needed concepts in the theory chapter (see section Theory Chapter). There you will find all the references to the relevant sources.

Definition of the Problem

✎🔗

In this chapter we try to explain our problem in a more scientific way and want to show all needed steps to fullfill the superordinate functionality of “avoiding obstacles”.

The only input is a RGB colored image, taken by a monocular camera (only one camera). The input image could look as Figure 14.1.

Sample image including some obstacles

With this information given, we want to find out whether an obstacle is in our way or not. If so, we want to either stop or adapt the trajectory to pass without crashing into the obstacle. This information is then forwarded as an output to the controllers who will process out commands and try to act accordingly.

Therefore one of the first very important decisions was to separate the detection and reaction parts of our saviors pipeline. This decision allowed us to divide our work efficiently, to start right away and is supposed to ensure a wide range of flexibility in the future by making it possible to easily replace, optimize or work on one of the parts separately (either the obstacle avoidance strategies or obstacle detection algorithms). Of course it also includes having to define a clear, reasonable interface in between the two modules, which will later be explained in detail.

You can have a look in our Preliminary Design Document and Intermediate Report to see how we defined the following topics in the beginning: The problem statement, our final objective, the underlying assumptions we lean on and the performance measurement to quantitatively check the perfomance of our algorithms. For the most part, it worked out to adhere to this document but for sake of completeness we will shortly repeat them again in the following for each of the two submodules.

Part 1: Computer Vision - Description

✎🔗

In principle we wanted to use the camera image only to reach the following:

  1. Detect the obstacles in the camera image
  2. Viusalize them in the camera image for tuning parameters and optimizing the code
  3. Give the 3D coordinates of every detected obstacle in the real world
  4. Give the size of every detected obstacle in the form of a radius around the 3D coordinate
  5. Label each obstacle if it’s inside or outside the lane boundaries (e.g. for the purpose of not stopping in a curve)
  6. Visualize them as markers in the 3D world (rviz)

Since every algorithm has its limitations, we made the following assumptions:

  • Obstacles are only yellow duckies and orange cones
  • Calibrated camera including intrinsics and extrinsics

Those assumptions changed slightly since the Preliminary Design Document because we are now also able to detect duckies on the middle line and in intersections.

It was our aim to reach the maximum within these specified limits. Therefore our goal was not only the detection and visualization in general but we also wanted to reach a maximum in robustness with respect to changes in:

  • Obstacle size
  • Obstacle color (within orange, and yellow to detect different traffic cones and duckies)
  • Illumination

For evaluating the performance, we used the following metrics, evaluated under different light conditions and different velocities (static and in motion):

  • Percentage of correctly classified obstacles on our picture datasets
  • Percentage of false positives
  • Percentage of missed obstacles

An evaluation of our goals and the reached performance can be found in the Performance Evaluation section.

Our approach is simply based on analysing incoming pictures for obstacles and trying to track them to make the algorithm more robust against outliers. Since we only rely on the monocular camera, we do not have any direct depth information given. In theory, it would be possible to estimate the depth of each pixel through some monocular visual odometry algorithm considering multiple consecutive images. However this would be extremely computationally expensive. The large amount of motion blur in our setup, a missing IMU (for estimating the absolute scale) further argue against such an approach. In our approach we use the extrinsic calibration to estimate the position of the given obstacles. The intuition behind that is that it is possible to assume that all pixels seen from the camera belong to the ground plane (except for obstacles which stand out of it) and that the Duckikebot’s relative position to this ground plane stays constant. Therefore you can assign a real world 3D coordinate to every pixel seen with the camera. For more details refer to the section below.

The final output is supposed to look as Figure 14.2.

Final output image including visualization of detected obstacles

Part 2: Avoidance in Real World - Description

✎🔗

With the from Part 1 given 3D position, size and the labelling whether the object is inside the lane boundaries or not, we wanted to reach the final objectives:

  1. Plan path around obstacle if possile (we have to stay within our lane)
  2. If this is not possible, simply stop

The assumptions for correctly reacting to the previously detected obstacles are:

  • Heading and position relative to track given
  • “The Controllers” are responsible for following our trajectory
  • Possibility to influence vehicle speed (slow down, stop)

As we now know, the first assumption is normally not fulfilled. We describe in the functionality section why this comes out to be a problem.

For measuring the performance we used:

  • Avoid/hit ratio
  • Also performed during changing light conditions

Contribution / Added Functionality

✎🔗

Software Architecture

✎🔗

In general we have four interfaces which had to be created throughout the implementation of our software:

1. At first, we need to recieve an incoming picture which we want to analyse. As our chosen approach includes filtering for specific colors, we are obviously dependent on the lighting conditions. In a first stage of our project, we nevertheless simply subscribed to the raw camera image because of the considerable expense of integrating the Anti Instagram Color Transformation and since the Anti Instagram team also first had to further develop their algorithms. During our tests we quickly recognized that our color filtering based approach would always have some troubles if we don’t compensate for the lighting change. Therefore, in the second part of the project we closely collaborated with the Anti Instagram team and are now subscribing to a color corrected image provided by them. Currently, to keep computational power on our Raspberry Pi low, the corrected image is published at 4Hz only and the color transformation needs at most 0.2 seconds.

2. The second part of our System Integration is the internal interface between the object detection and avoidance part. The interface is defined as a PoseArray which has the same timestamp as the picture from which the obstacles have been extracted. This Array, as the name already describes, is made up of single poses. The meaning of those are the following:

The position x and y describe the real world position of the obstacle which is in our case the center front coordinate of the obstacle. Since we assume planarity, the z coordinate of the position is not needed. That is why we are using this z coordinate to describe the radius of the obstacle.

Furthermore a negative z coordinate shows that there is a white line in between us and the obstacle which indicates that it is not dangerous to us since we assume to always having to stay in the lane boundaries. Therefore this information allows us to not stop if there is an obstacle behind a turn.

As for the scope of our project, the orientation of the obstacles is not really important, we use the remaining four elements of the Pose Message to pass the pixel coordinates of the bounding box of the obstacle seen in the bird view. This is not needed for our “Reaction” module but allows us to implement an efficient way of visualisation which will be later described in detail. Furthermore, we expect our obstacle detection module to add an additional delay of about max. 0.3s.

3. The third part is the interface between our obstacle avoidance node and the Controllers. The obstacle avoidance node generates an obstacle avoidance pose array and obstacle avoidance active flag.

The obstacle avoidance pose array is the main interface between the Saviors and the group doing lane control. We use the pose array to transmit d_ref (target distance to middle of the lane) and v_ref (target robot speed). The d_ref is our main control output which enables us to position the robot inside the lane and therefore to avoid objects which are placed close the laneline on the track. Furthermore v_ref is used to stop the robot when there is an unavoidable object by setting the target speed to zero.

The flag is used to communicate to the lane control nodes when the obstacle avoidance is activated which then triggers d_ref and v_ref tracking.

4. The fourth part is an optional interface between the Duckiebot and the user’s personal Laptop. Especially for the needs of debugging and infering what is going on, we decided to implement a visualisation node which can visualize on the one hand the input image including bounding boxes around all the objects which were classified as obstacles and furthermore this node can output the obstacles as markers which can be displayed in rviz.

In the following (Figure 14.3) you find a graph which summarises our software packages and gives a brief overview.

Module overview 'The Saviors'

Part 1: Computer Vision - Functionality

✎🔗

Let’s again have a look on the usual incoming camera picture in Figure 14.1.

In the very beginning of the project, like the previous implementation in 2016, we tried to do the detection in the normal camera image but we tried to optimize for more efficient and general obstacle descriptors. Due to the specifications of a normal camera, lines which are parallel in the real world are in general not parallel any longer and so the size and shape of the obstacles are disturbed (elements of the same size appear also larger in the front than in the back). This made it very difficult to reliably differentiate between yellow ducks and line segments. We tried several different approaches to overcome this problem, namely:

  • Patch matching of duckies viewed from different directions
  • Patch matching with some kind of an ellipse (because line segments are supposed to be square)
  • Measuring the maximal diameter
  • Comparing the height and the width of the objects
  • Taking the pixel volume of the duckies

Unfortunately none of the described approaches provided a sufficient performance. Also a combination of them didn’t make the desired impact. All metrices which are somehow associated with the size of the object just won’t work because duckies further away from the duckiebot are simply a lot smaller than the one very close to the Duckiebot. All metrices associated with the “squareness” of the lines were strongly disturbed by the ocurring motion blur. This makes finding a general criterion very difficult and made us think about changing the approach entirely.

Therefore we developed and came up with the following new approach!

Theoretical Description

✎🔗

In our setup, through the extrinsic camera calibration, we are given a mapping from each pixel in the camera frame to a corresponding real world coordinate. It is important to mention that this transformation assumes all seen pixels in the camera frame to lie in one plane which is in our case the ground plane/street. Our approach exactly exploits this fact by transforming the given camera image into a new, bird’s view perspective which basically shows one and the same scene from above. Therefore the information provided by the extrinsic calibration is essential for our algorithm to work properly. In Figure 14.4 you can see the newly warped image seen from the bird’s view perspective. This is one of the most important steps in our algorithm.

Image now seen from the bird's view perspective

This approach has already been shown by Prof. Davide Scaramuzza (UZH) and some other papers and is referred as Inverse Perspective Mapping Algorithm. (see: [32],[33],[34])

What stands out, is that the lines which are parallel in the real world are also parallel in this view. Generally in this “bird’s” view, all objects which really belong to the ground plane are represented by their real shape (e.g. the line segments are exact rectangles) while all the objects which are not on the ground plane (namely our obstacles) are heavily disturbed in this top view. This top view is roughly keeping the size of the elements on the ground whereas the obstacles are displayed a lot larger.

The theory behind the calculations and why the objects are so heavily distorted can be found in the Theory Chapter.

Either way, we take advantage of this property. Given this bird’s view perspective, we still have to extract the obstacles from it. To achieve this extraction, we first filter out everything except for orange and yellow elements, since we assumed that we only want to detect yellow duckies and orange cones. To simplify this step significantly, we transform the obtained color corrected images (provided by the Anti Instagram module) to the HSV color space. We use this HSV color space and not the RGB space because it is much easier to account for slightly different illuminations - which of course still exist since the performance of the color correction is logically not perfect - in the HSV room compared to RGB. For the theory behind the HSV space, please refer to our appropriate Theory Chapter.

After this first color filtering process, there are only objects remaining which have approximately the colors of the expected obstacles. For the purpose of filtering out the real obstacles from the bunch of all the remaining objects which passed the color filter, we decided to do the following: We segment the image of the remaining objects, i.e. all connected pixels in the filtered image are getting the same label such that you can later analyse the objects one by one. Each number then represents an obstacle. For the process of segmentation, we used the following algorithm. (see [35])

Given the isolated objects, the task remains to finally decide which objects are considered obstacles and which not. In a first stage, there is a filter criterion based on a rotation invariant feature, namely the two eigenvalues of the inertia_tensor of the segmented region when rotating around its center of mass. (see [36])

In a second stage, we apply a tracking algorithm to reject the remaining outliers and decrease the likelihood for misclassifications. The tracker especially aims for objects which passed the first stage’s criterion by a small margin.

For further informations and details about how we perform the needed operations, please refer to the next chapter.

The final output of the detection module is the one we showed in Figure 14.2.

Actual Implementation

✎🔗

Now we want to go more into detail how we implemented the described steps.

In the beginning we again start from the picture you can see in Figure 14.1. In our case this is now the corrected image coming out form the image_transformer_node and was implemented by the anti instagram group. We then perform the follwing steps:

1. In a first step we crop this picture to make our algorithm a little bit more efficient and due to our limited velocities, it makes no sense to detect obstacles which are not needed to be taken into consideration by our obstacle avoidance module. However, we do not simply crop the picture by a fixed amount of pixels, but we use the extrinsic calibration to neglect all the pixels which are farther away than a user defined threshold, which is at the moment at 1.7 meters. So the amount of pixels which are neglected are different for every Duckiebot and depend on the extrinsic calibration. The resulting image can be seen in Figure 14.5. The calculations to find out where You have to cut the image are quite simple (note that it still bargains for homogeneous coordinates):

$$ p_{camera} = H^{-1}P_{world} $$

Cropped image

2. Directly detecting the obstacles from this cropped input image failed for us due to the reasons descibed above. That is why the second step is to perform the transformation to the bird’s view perspective. For transforming the image, we first use the corners of the cropped image and transorm it to the real world. Then we scale the real world coordinates to pixel coordinates, so that it will have a width of 640 pixels afterwards. For warping all of the remaining pixels with low artifacts we use the function cv2.getPerspectiveTransform(). The obtained image can be seen in Figure 14.4.

3. Then we transform the given RGB picture into the HSV colorspace and apply the yellow and orange filter. While a HSV image is hardly readable for humans, it is way better to filter for specific colors. The obtained pictures can be seen in Figure 14.6 and Figure 14.7. The color filter operation is performed by the cv2 function cv2.inRange(im_test, self.lower_yellow, self.upper_yellow) where lower_yellow and upper_yellow are the thresholds for yellow in the HSV color space.

Yellow filtered image
Orange filtered image

4. Now there is the task of segmenting/isolating the objects which remained after the color filtering process. At the beginning of the project we therefore implemented our own segmentation algorithm which was however more inefficient and led to an overall computational load of 200% CPU usage and a maximum frequency of our whole module of about 0.5 Hz only. By using the scikit-image module which provides a very efficient label function, the computational efficiency could be shrunk considerably to about 70% CPU usage and allows the whole module to run at up to 3 Hz. It is important to remember that in our implementation the segmentation process is the one which consumes the most power. The output after the segmentation is the one in Figure 14.8, where the different colors represent the different segmented objects.

Segmented image

5. After the segmentation, we analyse each of the objects separately. At first there is a general filter which ensures that we are neglecting all the objects which contain less than a user influenced threshold of pixels. Since as mentioned above, the homographies of all the users are different, the exact amount of pixels, an object is required to have, is again scaled by the individual homography. This is followed by a more in detail analysis which is color dependent. On the one hand there is the challenge to detect the orange cones reliably. Speaking about cones, the only other object that might be erroneously detected as orange are the stop lines. Of course, in general the goal should be to very reliably detect orange but as the light is about to change during the drive, we prepared to also detect the stop lines and being able to cope with them when they are erroneously detected. The other general challenge was that all objects that we have to detect can appear in all different orientations. Simply inferring the height and width of the segmented box, as we did it in the beginning, is obviously not a very good measure (e.g. in Figure 14.9 in the lower left the segmented box is square while the cone itself is not quadratic at all).

Bird's view with displayed obstacle boxes

That is why it is best to use a rotation invariant feature to classify the segmented object. In our final implementation we came up with using the two eigenvalues of the inertia tensor, which are obviously rotation invariant (when being ordered by their size). Being more specific about the detection of cones, when extracting the cone from Figure 14.8 it is looking like in Figure 14.10, while an erroneous detection of a stop line is looking like in Figure 14.11.

Segmented cone
Segmented stop line

Our filter criteria is now the ratio between the eigenvalues of the inertia tensor. This ratio is always by a factor of about 100 greater in case the object is a cone, compared to when we erroneously segment a red stop line. This criteria is very stable that is why there is no additional filtering needed to detect the cones.

If the segmented object is yellowish, things get a little more tricky as there are always many yellow objects in the picture, namely the middle lines. Line elements can be again observed under every possible orientation. Therefore the eigenvalues of the inertia tensor, which are as mentioned above rotation invariant, are again the way to go. In Figure 14.12 you can see a segmented line element and in Figure 14.13 again a segmented duckie.

Segmented middle line
Segmented duckie

As the labelled axis already reveal, they are of a different scale, but as we also got very small duckies, we had to choose a very small threshold. To detect the yellow duckies, the initial condition is that the first eigenvalue has to be greater than 20. This criteria alone however includes to sometimes erroneously detecting the lines as obstacles, that is why we implemented an additional tracking algorithm which works as follows: If an object’s first eigenvalue is greater than 100 pixels and it is detected twice - meaning in two consecutive images there is a object detected at roughly the same place - it is labelled as an obstacle. However, if an object is smaller or changed the size by more than 50% in the consecutive frames, then a more restrictive criteria is enforced. This more restrictive criterion states that we must have tracked this object for at least for 3 consecutive frames before being labelled as an obstacle. This criteria is working pretty well and a more thorough evaluation will be provided in the next section. In general those criteria help that the obstacles can be detected in any orientation. The only danger to the yellow detecting algorithm is motion blur, namely when the single lines are not separated but connected together by “blur”.

6. After analysing each of the potential obstacle objects, we decide whether it is an obstacle or not. If so, we continue to steps 7. and 8..

7. Afterwards, we calculate the position and radius of all of the obstacles. After segmenting the object we calculate the 4 corners (which are connected in Figure 14.14 to form the green rectangle). We defined the obstacle’s position as the midpoint of the lower line (this point surely lies on the ground plane). For the radius, we use the distance in the real world between this point and the lower right corner. This turned out to be a good approximation of the radius. For an illustration you can have a look at Figure 14.14.

Position and radius of the obstacle

8. Towards the end of the project we came up with one additional last step based on the idea that only obstacles inside the white lane boundaries are of interest to us. That is why for each obstacle, we look whether there is something white in between us and the obstacle. In Figure 14.15 you can see an example situation where the obstacle inside the lane is marked as dangerous (red) while the other one is marked as not of interest to us since it is outside the lane boundary (green). In Figure 14.16 you see the search lines (yellow) along which we search for white elements.

Classification if objects are dangerous or not
Search lines to infer if something white is in between

9. As the last step of the detection pipeline we return a list of all obstacles including all the information via the Posearray.

Part 2: Avoidance in Real World - Functionality

✎🔗

The Avoidance deals with drawing the right conclusions from the received data and forwarding it.

Theoretical Description

✎🔗

With the separation of the detection, an important part of the avoidance node is the interaction with the other work packages. We determined the need of getting information about the remaining Duckietown besides the detected obstacles. The obstacles need to be in relation to the track, in order to assess whether we have to stop, can drive around obstacles or if it is even already out of track. Due to other teams already working on the orientation within Duckietown, we deemed it best to not implement any further detections (lines, intersections etc.) in our visual perception pipeline. This saves similar algorithms being run twice on the processor. We decided to acquire the values of our current pose relative to the side lane, which is determined by the devel-linedetection group.

The idea was to make the system highly flexible. The option to adapt to following situations was deemed desirable:

  • Multiple obstacles. Different path planning in case of a possible avoidance might be required.
  • Adapted behavior if the robot is at intersections.
  • Collision avoidance dependent on the fleet status within the Duckietown. Meaning if a Duckiebot drives alone in a town it should have the option to avoid a collision by driving onto the opposite lane.

Obstacles sideways of the robot were expected to appear as the Duckietowns tend to be flooded by duckies. Those detections on the side as well as far away false positive detections should not make the robot stop. To prevent that, we intended on implementing a parametrized bounding box ahead of the robot. Only obstacles within that box would be considered. Depending on the certainty of the detections as well as the yaw-velocities the parametrization would be tuned.

The interface getting our computed desired values to impact the actual Duckiebot is handled by devel-controllers. We agreed on the usage of their custom message format, in which we send desired values for the lateral lane position and the longitudinal velocity. Our intention was to account for the delay of the physical system in the avoider node. Thus our planned trajectory will reach the offset earlier than the ideal-case trajectory would have to.

Due to above mentioned interfaces and multiple levels of goals we were aiming for an architecture which allows gradual commissioning. The intent was to be able to go from basic to more advanced for us as well as for groups in upcoming years. Those should be able to extend our framework and not have to rebuild it.

The logic shown in Figure 14.17 displays one of the first stages in the commissioning. Key is the reaction to the number of detected obstacles. Later stages will not trigger an emergency stop in case of multiple obstacle detections within the bounding box.

Logic of one of the First Stages in Commissioning

Our biggest concern were the added inaccuracies until the planning of the trajectory. Those include:

  • Inaccuracy of the currently determined pose
  • Inaccuracy of the obstacle detection
  • Inaccuracy of the effectively driven path aka. controller performance

To us the determination of the pose was expected to be the most critical. Our preliminary results of the obstacle detection seemed reasonably accurate. The controller could be tweaked that the robot would rather drive out of the track than into the obstacle. Though an inaccurate estimation of the pose would just widen the duckie artificially.

Devel-controllers did not plan on being able to intentionally leave the lane. Meaning the space left to avoid an obstacle on the side of the lane is tight making above uncertainties more severe.

We evaluated the option to keep track of our position inside the map. Given a decent accuracy of said position we’d be able to create a map of the detected obstacles. Afterwards - especially given multiple detections (also outside of the bounding box) - we could achieve a further estimation of our pose relative to the obstacles. This essentially would mean creating a SLAM-algorithm with obstacles as landmarks. We declared as out of scope given the size of our team as well as the computational constraints. The goal was to make use of a stable, continuous detection and in each frame react on it.

Actual Implementation

✎🔗

Interfaces

One important part of the Software is the handling of the interfaces, mainly to devel_controllers. For further informations on this you can refer to the Software Architecture Chapter.

Reaction

The obstacle avoidance part of the problem is handled by an additional node, called the obstacle_avoidance_node. The node uses two main inputs which are the obstacle pose and the lane pose. The obstacle pose is an input coming from the obstacle detection node, which contains an array of all the obstacles currently detected. Each array element consists of an x and y coordinate of an obstacle in the robot frame (with the camera as origin) and the radius of the detected object. By setting the radius to a negative value, the detection node indicates that this obstacle is outside the lane and should not be considered for avoidance. The lane pose is coming from the line detection node and contains among other unused channels the current estimated distance to the middle of the lane (d) as well as the current heading of the robot $\theta$. Figure 14.18 introduces the orientations and definitions of the different inputs which are processed in the obstacle avoidance node.

Variable Definitions seen from the Top

Using the obstacle pose array we determine how many obstacles need to be considered for avoidance. If the detected obstacle is outside the lane and therefore marked with a negative radius by the obstacle detection node we can ignore it. Furthermore, we use the before mentioned bounding box with tunable size which assures that only objects in a certain range from the robot are considered. As soon as an object within limits is inside of the bounding box, the obstacle_avoidance_active flag is set to true and the algorithm already introduced in Figure 14.17 is executed.

Case 1: Obstacle Avoidance

If there is only one obstacle in range and inside the bounding box, the obstacle avoidance code in the avoider function is executed. First step of the avoider function is to transform the transmitted obstacle coordinates from the robot frame to a frame which is fixed to the middle of the lane using the estimated measurements of $\theta$ and d. Doing this transformation allows us to calculate the distance of the object from the middle line. If the remaining space (in the lane (subtracted by a safety avoidance margin) is large enough for the robot to drive through we proceed with the obstacle avoidance, if not we switch to case 2 and stop the vehicle. Please refer to Figure 14.19

Geometry of described Scene

If the transformation shows that an avoidance is possible we calculate the d_ref we need to achieve to avoid the obstacle. This is sent to the lane control node and then processed as new target distance to the middle of the lane. The lane control node uses this target and starts to correct the Duckiebot’s position in the lane. With each new obstacle pose being generated this target is adapted so that the Duckiebot eventually reaches target position. The slow transition movement allows us to avoid the obstacle even when it is not visible anymore shortly before the robot is at the same level as the obstacle.

At the current stage, the obstacle avoidance is not working due to very high inaccuracies in the estimation of $\theta$. The value shows inaccuracies with an amplitude of 10°, which leads to wrong calculations of the transformation and therefore to misjudgement of the d_ref. The high amplitude of these imprecisions could be transformed to a uncertainty factor of around 3 which means that each object is around 3 times its actual size which means that even a small obstacle on the side of the lane would not allow a safe avoidance to take place. For this stage to work, the estimation of $\theta$ would need significant improvement.

Case 2: Emergency Stop

Conditions for triggering an emergency stop:

  • More than one obstacle in range
  • Avoidance not possible because the obstacle is in the middle of the lane
  • Currently every obstacle detection in the bounding box triggers an emergency stop due to the above reasons

If one of the above scenarios occurs, an avoidance is not possible and the robot needs to be stopped. By setting the target speed to zero, the lane controller node stops the Duckiebot. As soon as the situation is resolved by removing the obstacle which triggered the emergency stop, the robot can proceed with the lane following.

These tasks are then repeated at the frame rate of the obstacle detection array being sent.

Required Infrastructure - Visualizer

✎🔗

Especially when dealing with a vision based obstacle detection algorithm it is very hard to infer what is going on. One has to also keep the visual outputs low, to consume as less computing power as possible, especially on the Raspberry Pi. This is why we decided to not implement one single obstacle detection node, but effectively two of them, together with some scripts which should help to tune the parameters offline and to infer the number of false positives, etc.. The node which is designed to be run on the Raspberry Pi is our normal obstacle_detection_node. This should in general be run such that there is no visual output at all but that simply the PoseArray of obstacles is published through this node.

The other node, namely the obstacle_detection_visual_node is designed to be run on your own laptop which is basically visualising the information given by the posearray. There are two visualisations available. On the one hand there is a marker visualisation in rviz which shows the position and size of the obstacles. In here all the dangerous obstacles which must be considered are shown in red, whereas the non critical (which we think that they are outside the lane boundaries) are marked in green. On the other hand there is also a visualisation available which shows the camera image together with bounding boxes around the detected obstacles. Nevertheless, this online visualisation is still dependent on the connectivity and you can only hardly “freeze” single situations where our algorithm failed. That is why we also included some helpful scripts into our package. One script allows to thoroughly input many pictures and outputs them labelled together with the bounding boxes, while another one outputs all the intermediate steps of our filtering process which allows to fastly adapt e.g. the color thresholds which is in our opinion still the major reason for failure. More information on our created scripts can be found in our Readme on GitHub.

Recorded Logs

✎🔗

For being able to thorougly evaluate and tune our algorithms, we recorded various bags, which we uploaded to the Duckietown logs database.

Formal Performance Evaluation / Results

✎🔗

Evaluation of the Interface and Computational Load

✎🔗

In general as we are dealing with many color filters a reasonable color corrected image is the key to the good functioning of our whole module, but turned out to be the greatest challenge when it comes down to computational efficiency and performance. As described above we are really dependent on a color corrected image by the Anti Instagram module. Throughout the whole project we planned to use their continuous anti-instagram node which is supposed to compute a color transformation in fixed intervals of time. However, when it came down we acutally had to change this for the follwing reason: The continuous anti-instagram node, running at an update interval of 10 seconds, consumes a considerable amount of computing power, namely 80%. In addition to that, the image transformer node which is in fact transforming the whole image and currently running at 4 Hz needs another 74% of one kernel. If you now run those two algorithms combined with the lane-following demo which makes the vehicle move and combined with our own code which needs an additional 75% of computing power, our safety critical module could only run at 1.5Hz and resulted in poor behaviour.

Even if you increase the time interval in which the continuous anti-instagram node computes a new transformation there was no real improvement. That is why in our final setup we let the anti-instagram node once compute a reasonable transformation and then keep this one for the entire drive. Through this measure we were able to safe the 80% share entirely and this allowed our overall node to be run at about 3 Hz with introducing an additional maximal delay of about 0,3 seconds. Nevertheless we want to point out that all the infrastructure for using the continuous anti instagram node in the future is provided in our package.

To sum up, the interface between our node and the Anti Instagram node was for sure developed very well and the collabroation was very good but when it came to getting the code to work, we had to take one step back to achieve good performance. That is why it might be reasonable to put effort into this interface in the future, to being able to more efficiently transform an entire image and to reduce the computational power consumed by the node which continuously computes a new transformation.

Evaluation of the Obstacle Detection

✎🔗

In general, since our obstacle classification algorithm is based on the rotational invariant feature of the eigenvalues of the inertia tensor it is completely invariant to the current orientation of the duckiebot and its position with respect to the lanes.

To rigorously evaluate our detection algorithm, we started off with evaluating static scenes, meaning the Duckiebot is standing still and not moving at all. Our algorithm performed extremely well in those static situations. You can place an arbitrary amount of obstacles, where the orientation of the respective obstacles does not matter at all, in front of the Duckiebot. In those situations and also combining them with changing the relative orientation of the Duckiebot itself, we achieved a false positive percentage of below 1% and we labelled all of the obstacles with respect to the lane boundaries correctly. The only static setup which is sometimes problematic is when we place the smallest duckies very close in front of our vehicle (below 4 centimeters), without approaching them. Then we sometimes cannot detect them. However this problem is mostly avoided during the dynamic driving, since we anyways want to stop earlier than 4 centimeters in front of potential obstacles. We are very happy with this static behaviour as in the worst case, if during the dynamic drive something goes wrong, you can still simply stop and rely upon the fact that the static performance is very good before continuing your drive. In the log chapter it is possible to find the corresponding logs.

This in return also implies that most of the misclassification errors during our dynamic drive are due to the effect of motion blur, assuming a stable color transformation provided by the anti instagram module. E.g. in Figure 14.20 two line segments in the background “blurred” together for two consecutive frames resulting in being labelled as an obstacle.

Obstacle Detector Error due to motion blur

Speaking more about of numbers, we took 2 duckiebots at a gain of around 0.6 and performed two drives at different days, so also at different lights and the results are the following: Evaluating each picture which will be given to the algorithm, we found out that on average, we detect 97% of all the yellow duckies in each picture. In terms of cones we detect about 96% of all cones in the evaluated frames. We consider these to be very good results as we have a very low rate of false positives (below 3%).

Date #correctly detected duckies #correctly detected cones #missed ducks #missed cones #false positive #false position
19/12/2017 423 192 14 8 9 45
Robot:Arki 3,2% 4% 1,4% 7,2%
Duration:82s
— —
21/12/2017 387 103 10 5 15 28
Robot:Dori 2,5% 4,4% 3% 5,7%
Duration:100s

When it comes to evaluating the performance of our obstacle classification with respect to classifying them as dangerous or not dangerous our performance is not as good as the detection itself, but we did also not put the same effort into it. As you can see in the table above, we have an error rate of above 5% when it comes to determining whether the obstalce’s position is inside or outside the lane boundaries (this is denoted as false position in the table above). We are especially encountering problems when there is direct illumination on the yellow lines which are very reflective and therefore appear whitish. Figure 14.21 shows such a situation where the current implementation of our obstacle classification algorithm fails.

Obstacle Detector Classification Error

Evaluation of the Obstacle Avoidance

✎🔗

Since at the current state we stop for every obstacle which is inside the lane and inside the bounding box, the avoidance process is very stable since it does not have to generate avoidance trajectories. The final performance on the avoidance is mainly relying on the placement of the obstacles:

1. Obstacle placement on a straight: If the obstacle is placed on a straight with a sufficient distance from the corner the emergency stop works nearly every time if the obstacle is detected correctly.

2. Obstacle in a corner: Due to the currently missing information of the curvature of the current tile the bounding box is always rectangular in front of the robot. This leads to problems if an obstacle is placed in a corner because it might enter the bounding box very late (if at all). Since the detection very close to the robot is not possible, this can lead to crashes.

3. Obstacles on intersection: These were not yet considered in our scope but still work if the detection is correct. It then behaves similar to case 1.

Furthermore there a few cases which can lead to problems independent of the obstacle placement: 1. Controller oscillations: If the lane controller sees a lot of lag due to high computing loads or similar its control sometimes start to oscillate. These oscillations lead to a lot of motion blur which can induce problems in the detection and shorten the available reaction time to trigger an emergency stop.

2. Controller offsets: The current size of the bounding box assumes that the robot is driving in the middle of the lane. If the robot is driving with an offset to the middle of the lane it can happen that obstacles at the side of the lane aren’t detected. This however rarely leads to crashes because then the robot is just avoiding the obstacle instead of stopping for it.

While testing our algorithms we saw successfull emergency stops in 10/10 cases for obstacles on a straight and in 3/10 cases for obstacles placed in corners assuming that the controller was acting normally. It is to be noted that the focus was lying on the reliable detections on the straights, which we intended to show on the demo day.

Future Avenues of Development

✎🔗

As already described above in the eval interface section, we think that there is still room for improving the interface between our code and the Anti Instagram module in terms of making the continouus anti instagram node as well as the image transformer node more computationally efficient. Another interesting thought which might be taken into consideration concerning this interface is the follwoing: As long as the main part of the anti instagram’s color correction is linear (which was in most of our cases sufficient), it might be reasonable to just adapt the filter values than to subscribe to a fully transformed image. This effort could save a whole publisher and subscriber and it is obvious that it is by far more efficient to transform a few filter values once than to transform every pixel of every incoming picture. Towards the end of our project we invested some time in trying to get this approach to work but as time was not enough we could not make it. We especially struggled to transform the orange filter values, while it worked for the yellow ones (BRANCH: devel-saviors-ai-tryout2). We think that if in the future one will stick to the current hardware this might be a very interesting approach, also for other software components such as the lane detection or any other picture related algorithms which are based on the concept of filtering colors.

Another idea of our team would be to exploit the transformation to the bird’s view also for other modules. We think that this approach might be of interest e.g. for extracting the curvature of the road or performing the lane detection from the rather more undistorted top view.

Another area of improvement would be to further develop our provided scripts to being able to automatically evaluate the performance of our entire pipeline. As you can see in our code description in github there is a complete set of scripts available which makes it easily possible to transform a bag of raw camera images to a set of pictures on which we applied our obstacle detector, including the color correction part of Anti Instagram. The only missing step left is an automatic detection whether the drawn box is correct and in fact around an object which is considered to be an obstacle or not.

Furthermore to achieve more general performance propably even adaptions in the hardware might be considered (see [37]) to tune the obstacle detection algorithm and especially its generality. We think that setting up a neural network might make it possible to release the restrictions on the color of the obstacles.

In terms of avoidance there would be possibilities to handle the high inacurracies of the pose estimation by relying on the lane controller to not leave the lane and just use a kind of closed loop control to avoid the obstacle (use the new position of the detected obstacle in each frame to securely avoid the duckie). Applying filters to the signals, especially the heading estimation, could further improve the behaviour. This problem was detected late in the development and could not be tested due to time constraints. Going further, having both the line and obstacle detection in the same algorithm would allow the direct information on how far away obstacles are from the track. We expect that this would increase the accuracy compared to computing each individually and bringing it together.

The infrastructure is in place to include new scenarios like obstacles on intersection or multiple detected obstacles inside the bounding box. If multiple obstacles are in proximity, a more sophisticated trajectory generation could be put in place to avoid these obstacles in a safe and optimal way.

Furthermore the avoidance in corners could be easily significantly improved if the line detection would estimate the curvature of the current tile which would enable adaptions to the bounding box oncorner tiles. If the pose estimation is significantly improved one could also implement an adaptive bounding box which takes exactly the form of the lane in front of the robot (see Figure 14.22)

Adaptive bounding box

Theory Chapter

✎🔗

Inverse Perspective Mapping / Bird’s View Perspective

✎🔗

The first chapter above introduced the rough theory which is needed for understanding the follwing parts. The important additional information that we exploited heavily in our approach is that in our special case we know the coordinate $Z_W$. The reason therefore lies within the fact that unlike in another more general usecase of a mono camera, we know that our camera will always be at height $h$ with repsect to the street plane and that the angle $\theta_0$ also always stays constant. (Figure 14.27)

Illustration of our fixed camera position [42]

This information is used in the actual extrinsic calibration such that in Duckietown, due to the assumption that everything we see should in general be on the road, we can determine the full real world coordinates of every pixel, since we know the coordinate $Z_W$ which uniquely defines the absolute scale and can therefore uniquely determine $\lambda$ and H! Intuitively this comes from the fact that we can just intersect the known ray direction (see Figure 14.24) with the known “gound plane”.

This makes it possible to project every pixel back into the “road plane” by computing for each available pixel: $$ \vec{P_W}=H^{-1} * \lambda * P_{pix} $$

This “projection back onto the road plane” is called inverse perspective mapping!

If you now visualize this “back” projection, you basically get the bird’s view since you can now map back every pixel in the image plane to a unique place on the road plane.

The only trick of this easy maths is that we exploited the knowledge that everything we see in the image plane is in fact on the road and has one and the same z-coordinate. You can see that the original input image Figure 14.28 is nicely transformed into the view from above where every texture and shape is nicely reconstructed if this assumption is valid Figure 14.29. You can especially see that all the yellow line segments in the middle of the road roughly have the same size in this bird’s view Figure 14.29 which is very different if you compare it to the original image Figure 14.28.

Normal incoming image without any obstacle
Incoming image without obstacle reconstructed in bird's view

The crucial part is now what happens in this bird’s view perspective, if the camera sees an object which is not entirely part of the ground plane, but stands out. These are basically obstacles we want to detect. If we still transform the whole image to the bird’s view, these obstacles which stand out of the image plane get heavily disturbed. Lets explain this by having a look at Figure 14.30.

Illustration why obstacle standing out of ground plane is heavily disturbed in bird's view, modified: [42]

The upper picture in Figure 14.30 depicts the real world situation, where the cone is standing out ot the image plane and therefore the tip is obviously not at the same height as the ground plane. However, as we still have this assumption and as stated above intuitively intersect the ray with the ground plane, the cone gets heavily disturbed and will look like the lower picture in Figure 14.30 after performing the inverse perspective mapping. From this follows that if there are any objects which DO stand out of the image plane then in the inverse perspective you basically see their shape being projected onto the ground plane. This behaviour can be easily exploited since all of these objects are heavily disturbed, drastically increase in size and can therefore be easily separated from the other objects which belong to the ground plane.

Let’s have one final look at an example in Duckietown. In Figure 14.31 you see an incoming picture seen from the normal camera perspective, including obstacles. If you now perform the inverse perspective mapping, the picture looks like Figure 14.32 and as you can easily see, all the obstacles, namely the two yellow duckies and the orange cone which stand out of the ground plane are heavily disturbed and therefore it is quite easy to detect them as real obstacles.

Normal situation with obstacles in Duckietown seen from Duckiebot perspective
Same situation seen from bird's perspective

HSV Color Space

✎🔗

Introduction and Motivation

✎🔗

The “typical” color model is called the RGB color model. It simply uses three numbers for the amount of the colors red, blue and green. It is an additive color system, so we can simply add two colors to produce a third one. Mathematically written it looks as follows and shows the way of how we deal with producing new colors:

$$ \left( \begin{array}{c} r_{res} \\ g_{res} \\ b_{res} \end{array} \right) = \left( \begin{array}{c} r_{1} \\ g_{1} \\ b_{1} \end{array} \right) + \left( \begin{array}{c} r_{2} \\ g_{2} \\ b_{2} \end{array} \right) $$

If the resulting color is white, the two colors 1 and 2 are called to be complementary (e.g. this is the case for blue and yellow).

This color system is very intuitive and is oriented on how the human vision perceives the different colors.

The HSV color space is an alternative representation of the RGB color model. On this occasion HSV is an acronym for Hue, Saturation and Value. It is not so easy summable as the RGB model and it is also hardly readable for humans. So the big question is: Why should we transform our colors to the HSV space? Does it derive a benefit?

The answer is yes. It is hardly readable for humans but it is way better to filter for specific colors. If we look at the definition openCV gives for the RGB space, the higher complexity for some tasks becomes obvious:

In the RGB color space all “the three channels are effectively correlated by the amount of light hitting the surface”, so the color and light properties are simply not separated. (see: [44])

Expressed in a more simpler way: In the RGB space the colors also influence the brightness and the brightness influences the colors. However, in the HSV space, there is only one channel - the H channel - to describe the color. The S channel represents the saturation and H the intensity. This is the reason why it is super useful for specific color filtering tasks.

The HSV color space is therefore often used by people who try to select specific colors. It corresponds better to how we experience color. As we let the H (Hue) channel go from 0 to 1, the colors vary from red through yellow, green, cyan, blue, magenta and back to red. So we have red values at 0 as well as at 1. As we vary the S (saturation) from 0 to 1 the colors simply vary from unsaturated (more grey like) to fully saturated (no white component at all). Increasing the V (value) the colors just become brighter. This color space is illustrated in Figure 14.33. (see: [45])

Illustration of the HSV Color Space [45]

Most systems use the so called RGB additive primary colors. The resulting mixtures can be very diverse. The variety of colors, called the gamut, can therefore be very large. Anyway, the relationship between the constitutent amounts of red, green, and blue lights is unintuitive.

Derivation

✎🔗

The HSV model can be derived using geometric strategies. The RGB color space is simply a cube where the addition of the three color components (with a scale form 0 to 1) is displayed. You can see this on the left of Figure 14.34.

Comparison between the two colors spaces [47]

You can now simply take this cube and tilt it on its corner. We do it this way so that black rests at the orgin whereas white is the highest point directly above it along the vertical axis. Afterwards you can just measure the hue of the colors by their angle around the vertical axis (red is denoted as 0°). Going from the middle to the outer parts from 0 (where the grey like parts are) to 1 determines the saturation. This is illustrated in Figure 14.35.

'Cutting the cube' [48]

The definitions of hue and chroma (proportion of the distance from the origin to the edge of the hexagon) amount to a geometric warping of hexagons into circles (for more informations see: [48]). Each side of the hexagon is mapped linearly onto a 60° arc of the circle. This is visualized in Figure 14.36.

Warping hexagons to circles [48]

For the value or lightness there are several possibilities to define an appropriate dimension for the color space. The simplest one is just the average of the three components, which is nothing else then the vertical height of a point in our tilted cubic. For this case we have:

$$ I = 1/3 * (R + G + B) $$

For another definition the value is defined as the largest component of a color. This places all three primaries and also all of the “secondary colors” (cyan, magenta, yellow) into a plane with white. This forms a hexagonal pyramid out of the RGB cube. This is called the HSV “hexcone” model and is the common one. We get:

$$ V = max(R, G, B) $$

(see: ([48]))

In Practice

✎🔗

1. Form a hexagon by projecting the RGB unit cube along its pincipal diagonal onto a plane.

First layer of the cube (left) and flat hexagon (right) [52]

2. Repeat projection with smaller RGB cube (subtract 1/255 in length of every cube) to obtain smaller projected hexagon. Like this a HSV hexcone is formed by stacking up the 256 hexagons in decreasing order of size.

Stacking hexagons together [52]

Then the value is again defined as:

$$ V = max(R, G, B) $$

3. Smooth edges of hexagon to circles (see previous chapter).

Application

✎🔗

One nice example of the application of the HSV color space can be seen in Figure 14.39.

Image on the left is original. Image on the right was simply produced by rotating the H of each color by -30° while keeping S and V constant [48]

It just shows how simple color manipulation can be performed in a very intuitive way. We can turn many different applications to good account using this approach. As you have seen, color filtering also simply becomes a threshold query.

Because of mathjax bug

PDD - Parking

✎🔗

Parking: intermediate report

✎🔗

Explicit Coordination: preliminary report

✎🔗

Explicit Coordination: intermediate Report

✎🔗

Explicit coordination: final report

✎🔗

The final result

✎🔗

Video of the final result:

The video is at https://vimeo.com/257762830.

Explicit coordination's video

To reproduce the results please see the operation manual .

Mission and Scope

✎🔗

Our mission is to coordinate the intersection navigation safely and cleverly through explicit communication.

Motivation

✎🔗

Duckietowns are complex systems where the traffic situations of a real city should be emulated. These towns contain three- and four-way intersections: the Duckiebots should be able to navigate them without crashing into each other and this requires a clever coordination scheme. Intersections represent a key element of a smooth city navigation.

There are two ways of coordinating Duckiebots: - Using a traffic light, - Using a communication protocol between the vehicles.

Hence, we aim to have both a centralised and a decentralised solution as well as an integration of the two. While the centralised solution boils down to understand the signal emitted by a referee (i.e., a traffic light), the decentralised coordination scheme should allow the Duckiebots to operate on their own, i.e., to communicate between each other and to take decisions without any external help.

Existing solution

✎🔗

A prior implementation for intersection coordination was already available from the 2016’s MIT class. The principle was simple: When a Duckiebot comes at an intersection and stops at the red line. Then, In the case with a traffic light, the Duckiebot detects the frequency at which the traffic light is blinking and acts based on the road rules. In the case without traffic light, the Duckiebot detects the frequency at which the other bots are blinking and adjusts its emitted frequency depending on its state. From an implementation perspective, the distinction was made a priori, i.e., the Duckiebots were not making use of the information coming from the Apriltags detection and therefore not were able to navigate systems with both types of intersection.

From the description above, we can distinguish two modules:

  • LED emission and detection,
  • Coordination based on the detected signals.

For the emission, three signals can be produced: each signal is encoded with a specific color and frequency. While Duckiebots are designed to recognise only frequencies, color are used to allow humans to easily understand the signals emitted by the Duckiebots. The signals represent the states: negotiation (and in which phase of the negotiation it is) or navigation of the intersection.

For the detection, the position and frequency of blinking LEDs are registered.

For the coordination, with the assumption that each vehicle can only see other vehicles on its right but not on its left, the Duckiebot yields its position if the only visible car is on the right, otherwise the Duckiebot waits (light green or red) or crosses (yellow light).

Opportunity

✎🔗

The existing solution had essentially two drawbacks:

  • The overall success rate was about 50%: The algorithm for LED-detection, and/or coordination failed, leading to a potential crash of the Duckiebots.
  • In case of success, the LED-detection and/or coordination algorithms required an average of four minutes to clear an intersection with four Duckiebots. This results in an extremely slow process, which would block a city with dozens of Duckiebots.

Although the solution was problematic, it still gave us some important intuitions on how to solve the problem.

First of all, using a LED-communication protocol is a brilliant idea to let the Duckiebots communicate with each other. Since the previous communication algorithm had the only disadvantage of being slow (also because of several bugs), we started by re-thinking the coordination algorithm. The existing implementation for the coordination was rather complex and articulated, resulting in confused strategies which led to the failure rate of 50%. In order to develop a simpler and lighter algorithm we took inspiration from an existing media access control protocol (MAC): the so called Carrier Sense Multiple Access (CSMA, https://en.wikipedia.org/wiki/Carrier-sense_multiple_access). This algorithm gave us the basic idea behind our strategy and allowed us to have a lighter protocol.

In the second place, we re-designed the LED-detection/-interpreter to be more efficient and robust based on the detection of blobs and not only on the detection of frequencies.

Lastly, we wanted to have a demo that could deal with both intersections, i.e. with and without a traffic light, as opposed to the two available demos from MIT 2016’s class.

Definition of the problem

✎🔗

Contribution / Added functionality

✎🔗

Formal performance evaluation / Results

✎🔗
Performance Evaluation
Situation Performance measure Required Obtained
One Duckiebot at the intersection Clearing time 60s 12s
One Duckiebot at the intersection Success rate 90% 98%
Two Duckiebots at the intersection Clearing time 60s 25s
Two Duckiebots at the intersection Success rate 80% 89%
Three Duckiebots at the intersection Clearing time 60s 50s
Three Duckiebot at the intersection Success rate 70% 89%
One Duckiebot at a traffic light type intersection Clearing time 60s 25s
One Duckiebot at a traffic light type intersection Success rate 90% 92%

Failures were mainly caused by the following reasons: - Duckiebots detecting the wrong sign (i.e., expecting a traffic light instead of a normal intersection). - Blobs not properly detected. This is mainly due of failures in the parameters and in the camera calibration. - Duckiebots crashing because of poor intersection navigation.

Future avenues of development

✎🔗

There is room for improvement for the coordination part of this project. Our approach, in the case of an intersection without traffic light, prioritises robustness rather than efficiency (in some cases all the Duckiebots at an intersection could turn off and restart the whole protocol again) and it is easy to imagine a scenario with an improved efficiency (tradeoff with complexity).

An idea would be to encode in the signal also the intentions of the Duckiebot and, by doing so, allow multiple Duckiebots to navigate the intersection at the same time if their directions are compatible. In fact, if two Duckiebots wanted to go straight they could move at the same time. The clearing time could also be reduced in the case of an intersection with traffic light if the latter was able to see where the vehicles are (prevent the light to turn green in a direction where no vehicle is waiting to cross).

Because of mathjax bug

Implicit Coordination: preliminary report

✎🔗

Implicit Coordination: intermediate report

✎🔗

Implicit Coordination: final report

✎🔗

Fleet Planning: Preliminary Report

✎🔗

Fleet Planning: Intermediate Report

✎🔗

This document describes system that the Fleet-planning team is planning to implement. Part 1 describes the interfaces to other parts of the system. I.e. How it communicates with all of them. In Part 2 a plan for the demo and evaluation is presented. And finally part 3 focuses on data collection, annotation and analysis. As for this project not much annotated data is used, part 3 is rather short.

Fleet Planning: final Report

✎🔗

Transferred Lane following

✎🔗

This is the description of transferred lane following demo.

Wheels calibration completed.wheel calibration

Camera installed in the right position.

Joystick demo has been successfully launched.Joystick demo

PyTorch installed on duckiebot and laptop. (On duckiebot, you can either build from source(link), or download the pytorch-0.2 wheel file we built)

Video of expected results

✎🔗

link 1 of lane following link 2 of lane following

Duckietown setup notes

✎🔗

A duckietown with white and yellow lanes. No obstacles on the lane.

Duckiebot setup notes

✎🔗

Make sure the camera is heading ahead. Tighten the screws if necessary.

Pre-flight checklist

✎🔗

Check: turn on joystick.

Check: Enough battery of the duckiebot.

Demo instructions

✎🔗

Here, give step by step instructions to reproduce the demo.

Step 1: On duckiebot, in /DUCKIERTOWN_ROOT/ directory, run command:

duckiebot `roslaunch deep_lane_following lane_following.launch`

Wait a while so that everything has been launched. Press R1 to start autonomous lane following. Press L1 to switch to joystick control.

The following is the same as demo-lane-following: Empirically speaking, no duckiebot will successfully run the demo for its first time. Parameter tuning is a must. The only two parameters that you can modify is the gain and trim. The parameter pair which makes your bot go straight will unlikely work for the lane following due to the current controller design. Facts show that a gain ranging from 0.5 to 0.9, as long as paired with a suitable trim, will all work on this demo. Start with your parameter pair obtained from wheel calibration. Increase gain for higher speed. Increase trim to horizontally move the bot to the center of the lane. Decrease will do the inverse.

Step 2: On laptop, make sure ros environment has been activated, run command:

laptop `rqt`

In rqt, the images can be visualized are /(vehicle_name)/camera_node/image/compressed

Troubleshooting

✎🔗

Contact Chip Schaff or Ruotian Luo(TTIC) via Slack if any trouble occurs.

Because of mathjax bug

Transfer Learning in Robotics

✎🔗

Understanding transfer learning and the domain randomization technique for transfer learning.

This unit introduces the concept of Transfer Learning and how it can be applied to Robotics.

Transfer Learning Definition

✎🔗

Transfer learning is a subfield of machine learning that focuses on using knowledge gained while solving one problem to solve a related problem.

PDD - Supervised-learning

✎🔗

Supervised Learning: intermediate report

✎🔗

PDD Neural Slam

✎🔗

PDD - Visual Odometry

✎🔗

Visual Odometry Project

✎🔗

Here we briefly describe the theory behind the model in the visual odometry project. The discussion begins with a review of epipolar geometry and a description of the depth image-based rendering problem, then moves to the description of the deep learning model used.

Deep Visual Odometry ROS Package

✎🔗

PDD - Anti-Instagram

✎🔗

PDD - Distributed Estimation

✎🔗

Demo instructions Fleet Communications

✎🔗

fix spelling and grammar

This is the description of a communication setup between multiple Duckiebots.

At least two Duckiebots in configuration DB17-w or higher.

One additional wireless adapter per Duckiebot and laptop. (e.g. TP-Link TL-WN822N or TL-WN821N).

A laptop.

Duckietown setup notes

✎🔗

For this Demo, no Duckietown is needed.

For this demo, additional wireless adapters are needed that allow mesh networking (e.g. TP-Link TL-WN822N or TL-WN821N).

Pre-flight checklist

✎🔗

This pre-flight checklist describes the steps that ensure that the installation and demo will run correctly:

Check: The additional Wifi adapter is installed and works.

$ sudo apt-get install build-essential linux-headers-generic dkms git
$ git clone https://github.com/Mange/rtl8192eu-linux-driver.git
$ sudo dkms add ./rtl8192eu-linux-driver
$ sudo dkms install rtl8192eu/1.0

Check: Duckiebots have sufficient battery charge.

Demo setup

✎🔗

Some packages are needed to enable the communication beween the Duckiebots, namely Protobuf, ZeroMQ and B.A.T.M.A.N.

To install them, ssh into the Duckiebots and source the environment

$ cd duckietown
$ source environment.sh

pull the necessary files from devel-distributed-est-master.

Then find the name of the wifi interface you want to use with iwconfig. (eg. wlx7c8bca1120e0).

$ iwconfig

Next specify a static IP adress and subnet and write it on a piece of paper, be carefull to not use the same IP on two bots. However, the subnet should stay the same on all bots. (eg. 192.168.15.38/24)

Change to dependecie directory

$ cd ~/duckietown/catkin_ws/src/30-localization-and-planning/fleet_messaging/dependencies

and install everything with one handy script!

$ ./install_fleet_messaging <wifi-iface> <ipaddr>

Now you need to alter your network config, for this open the interfaces file:

$ sudo vim /etc/network/interfaces

Change all four instances of wlan0 to wlan1.

After a reboot you are ready to make your Duckiebots talk to each other.

Troubleshooting

✎🔗

It’s networking. If it doesn’t work try reinstalling while letting 99 duckies swim in the bathtub and lighting magic candles.

Demo failure demonstration

✎🔗

terminal_full_of_errors.avi

Because of mathjax bug

Transfer: preliminary design document

✎🔗

Packages - Infrastructure

✎🔗

to write

Because of mathjax bug
Because of mathjax bug

Package duckieteam

✎🔗

✎

Link to package on Github

✎

Andrea Censi (maintainer)

✎

Code for handling the DB of people and robots.

create-machines

✎🔗

This program creates the machines file, using the data contained in the scuderia database.

Run as follows:

$ rosrun duckieteam create-machines

create-roster

✎🔗

this program is unfinished

Because of mathjax bug
Because of mathjax bug

Package duckietown

✎🔗

✎

Link to package on Github

✎

Mack

✎

The duckietown meta package

Because of mathjax bug
Because of mathjax bug

Package duckietown_msgs

✎🔗

✎

Link to package on Github

✎

Mack

✎

Add a description of package duckietown_msgs in package.xml.

Because of mathjax bug
Because of mathjax bug

Package easy_algo

✎🔗

✎

Link to package on Github

✎

Andrea Censi (maintainer)

✎

A package to make it easy to abstract about algorithm configuration and create regression tests.

Package easy_logs

✎🔗

✎

Link to package on Github

✎

Andrea Censi (maintainer)

✎

The easy_logs packages manages the Duckietown logs.

It indexes them and has a simple interface to query the DB.

Tests should use the easy_logs DB to know what logs to work on.

Selector language

✎🔗

Here are some examples for the query language (Table 5.1).

Show all the Ferrari logs:

$ rosrun easy_logs summary vehicle:ferrari

All the logs of length less than 45 s:

$ rosrun easy_logs summary "length:<45"

All the invalid logs:

$ rosrun easy_logs summary "length:<45,valid:False"

All the invalid logs of length less than 45 s:

$ rosrun easy_logs summary "length:<45,valid:False"
Query language
expression example explanation
attribute:expr vehicle:ferrari Checks that the attribute [attribute of the object satisfies the expression in expr
>lower bound >10 Lower bound
<upper bound <1 Upper bound
expr1,expr2 >10,<20 And between two expressions
expr1+expr2 <5+>10 Or between two expressions
pattern *ferrari* Other strings are interpreted as wildcard patterns.

Automatic log download using download

✎🔗

The command-line program download downloads requires logs from the cloud.

The syntax is:

$ rosrun easy_logs download log name

For example:

$ rosrun easy_logs download 2016-04-29-dp3auto-neptunus-1

If the file 2016-04-29-dp3auto-neptunus-1.bag is not available locally, it is downloaded.

The database of URLs is at the file dropbox.urls.yaml in the package easy_node.

A typical use case would be the following, in which a script needs a log with which to work.

Using the download program we declare that we need the log. Then, we use find to find the path.

#!/bin/bash
set -ex

# We need the log to proceed
rosrun easy_logs download 2016-04-29-dp3auto-neptunus-1

# Here, we know that we have the log. We use `find` to get the filename.
filename=`rosrun easy_logs find 2016-04-29-dp3auto-neptunus-1`

# We can now use the log
vdir ${filename}

Browsing the cloud

✎🔗

How do you know which logs are in the cloud?

Run the following to download a database of all the logs available in the cloud:

$ make cloud-download

You can query the DB by using summary with the option --cloud:

$ rosrun easy_logs summary --cloud '*RCDP6*'

| #     Log name                                   date          length    vehicle name
| --    ---------------------------------------    ----------    ------    ------------
| 0     20160122-censi-ferrari-RCDP6-catliu        2016-02-28     194 s    ferrari
| 1     20160122-censi-ferrari-RCDP6-joe-wl        2016-02-27     196 s    ferrari
| 2     20160228-sanguk-setlist-RCDP6-sangukbo     2016-03-02     193 s    ferrari
| 3     20160122-censi-ferrari-RCDP6-eharbitz      2016-02-27     198 s    ferrari
| 4     20160122-censi-ferrari-RCDP6-teddy         2016-02-28     193 s    ferrari
| 5     20160122-censi-ferrari-RCDP6-jenshen       2016-02-29      83 s    ferrari

Then, you can download locally using:

$ rosrun easy_logs download 20160122-censi-ferrari-RCDP6-teddy

Once it is downloaded, the log becomes part of the local database.

Advanged log indexing and generation

✎🔗

Shuffle

✎🔗

expr/shuffle shuffles the order of the logs in expr.

Give me all the oreo logs, in random order:

$ rosrun easy_logs summary  vehicle:oreo/shuffle

Simple indexing

✎🔗

expr/[i] takes the i-th entry.

Give me the first log:

$ rosrun easy_logs summary  "vehicle:oreo/[0]"

Give me a random log; i.e. the first of a random list.

$ rosrun easy_logs summary  "vehicle:oreo/shuffle/[0]"

Complex indexing

✎🔗

You can use the exact Python syntax for indexing, including [a:], [:b], [a:b], [a:b:c], etc.

Give me three random logs:

$ rosrun easy_logs summary  "all/shuffle/[:3]"

Sorting

✎🔗

To implement.

Time indexing

✎🔗

You can ask for only a part of a log using the syntax:

expr/{start:stop}
expr/{start:}
expr/{:stop}

where start and stop are in time relative to the start of the log.

For example, “give me all the first 1-second intervals of the logs” is

all/{:1}

Cut the first 3 seconds of all the logs:

all/{3:}

Give me the interval between 30 s and 35 s:

all/{30:35}

How to set up the backend needed to use files from the cloud

✎🔗

This applies to any resource, not only logs.

First, put the file in the duckiedown-data-2017 directory on Dropbox.

This is the link.

You need to ask Liam/Andrea because only them have write access.

Then, get the public link address and put it in the file dropbox.urls.yaml in the package easy_node. Remember to have ?dl=1 instead of ?dl=0 in the url.

In the code, use the function require_resource():

from duckietown_utils import require_resource

zipname = require_resource('ii-datasets.zip')

The storage area is in ${DUCKIETOWN_ROOT}/cache/download.

If the file is already downloaded, it is not downloaded again.

(So, if the file changes, you need to delete the cache directory. The best practice is to change the filename every time the file changes.)

The function require_resource() returns the path to the downloaded file.

Also note that you can put any URL in the file dropbox.urls.yaml; but the convention is that we only link things to Dropbox.

Because of mathjax bug
Because of mathjax bug

Package easy_node

✎🔗

✎

Link to package on Github

✎

Andrea Censi (maintainer)

✎

easy_node is a framework to make it easier to create and document ROS nodes.

The main idea is to provide a declarative approach to describe:

  • The node parameters;
  • The node’s subscriptions;
  • The node’s publishers;
  • The node’s assumptions (contracts).

The user describes subscriptions, publishers, and parameters in a YAML file.

The framework then automatically takes care of:

  • Calling the necessary boilerplate ROS commands for subscribing and publishing topics.
  • Loading and monitoring configuration.
  • Create the Markdown documentation that describes the nodes.
  • Provide a set of common functionality, such as benchmarking and monitoring latencies.

Using easy_node allows to cut 40%-50% of the code required for programming a node. For an example, see the package line_detector2, which contains a re-implementation of line_detector using the new framework.

Transition plan: The plan is to first use easy_node just for documenting the nodes. Then, later, convert all the nodes to use it.

Benchmarking

✎🔗

EasyNode implements some simple timing statistics. These are accessed using the context object passed to the message received callbacks.

Here’s an example use, from line_detector2:

def on_received_image(self, context, image_msg):

    with context.phase('decoding'):
        ...

    with context.phase('resizing'):
        # Resize and crop image
        ...

    stats = context.get_stats()
    self.info(stats)

The idea is to enclose the different phases of the computation using the context manager phase(name).

A summary of the statistics can be accessed by using context.get_stats().

For example, this will print:

Last 24.4 s: received 734 (30.0 fps) processed 301 (12.3 fps) skipped 433 (17.7 fps) (59 %)
            decoding | total latency  25.5 ms | delta wall   20.7 ms | delta clock  20.7 ms
            resizing | total latency  26.6 ms | delta wall    0.8 ms | delta clock   0.7 ms
          correcting | total latency  29.1 ms | delta wall    2.2 ms | delta clock   2.2 ms
           detection | total latency  47.7 ms | delta wall   18.2 ms | delta clock  21.3 ms
    preparing-images | total latency  55.0 ms | delta wall    7.0 ms | delta clock   7.0 ms
          publishing | total latency  55.5 ms | delta wall    0.1 ms | delta clock   0.1 ms
          draw-lines | total latency  59.7 ms | delta wall    4.0 ms | delta clock   3.9 ms
    published-images | total latency  61.2 ms | delta wall    0.9 ms | delta clock   0.8 ms
pub_edge/pub_segment | total latency  86.3 ms | delta wall   24.7 ms | delta clock  24.0 ms

Package easy_regression

✎🔗

✎

Link to package on Github

✎

Andrea Censi (maintainer)

✎

A package to make it easy to abstract about algorithm configuration and create regression tests.

Package what_the_duck

✎🔗

✎

Link to package on Github

✎

Andrea Censi (maintainer)

✎

what-the-duck is a program that tests dozens of configuration inconsistencies that can happen on a Duckiebot.

The what-the-duck program

✎🔗

The proper usage of what-the-duck to debug an environment problem is the following sequence:

$ # open a new terminal
$ cd Duckietown root
$ git checkout master
$ git pull
$ source environment.sh
$ ./dependencies_for_duckiebot.sh # if you are on a Duckiebot
$ ./dependencies_for_laptop.sh    # if you are on a laptop
$ make build-clean
$ make build-catkin
$ ./what-the-duck

you have to do all the steps in the precise order.

✎

The telemetry is collected and available at this URL.

Because of mathjax bug

Packages - Teleoperation

✎🔗

to write

Because of mathjax bug
Because of mathjax bug

Package adafruit_drivers

✎🔗

✎

Link to package on Github

✎

Dmitry Yershov

Mack

✎

Add a description of package adafruit_drivers in package.xml.

These are the Adafruit drivers.

What is the original location of this package?

Because of mathjax bug
Because of mathjax bug

Package dagu_car

✎🔗

✎

Link to package on Github

✎

Dmitry Yershov

Shih-Yuan Liu

Mack

✎

Add a description of package dagu_car in package.xml.

Node forward_kinematics_node

✎🔗

(Generated from configuration forward_kinematics_node.easy_node.yaml.)

Missing node description in forward_kinematics_node.easy_node.yaml.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Node inverse_kinematics_node

✎🔗

(Generated from configuration inverse_kinematics_node.easy_node.yaml.)

Missing node description in inverse_kinematics_node.easy_node.yaml.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Node velocity_to_pose_node

✎🔗

(Generated from configuration velocity_to_pose_node.easy_node.yaml.)

Missing node description in velocity_to_pose_node.easy_node.yaml.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Node car_cmd_switch_node

✎🔗

(Generated from configuration car_cmd_switch_node.easy_node.yaml.)

Missing node description in car_cmd_switch_node.easy_node.yaml.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Node wheels_driver_node

✎🔗

(Generated from configuration wheels_driver_node.easy_node.yaml.)

Missing node description in wheels_driver_node.easy_node.yaml.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Node wheels_trimmer_node

✎🔗

(Generated from configuration wheels_trimmer_node.easy_node.yaml.)

Missing node description in wheels_trimmer_node.easy_node.yaml.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Because of mathjax bug
Because of mathjax bug

Package joy_mapper

✎🔗

✎

Link to package on Github

✎

Mack

✎

The joy_mapper package for duckietown. Takes sensor_msgs.Joy and convert it to duckietown_msgs.CarControl.

Package pi_camera

✎🔗

✎

Link to package on Github

✎

Andrea Censi

✎

Add a description of package pi_camera in package.xml.

Node camera_node_sequence

✎🔗

(Generated from configuration camera_node_sequence.easy_node.yaml.)

Camera driver, second approach.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Node camera_node_continuous

✎🔗

(Generated from configuration camera_node_continuous.easy_node.yaml.)

Camera driver.

✎

Parameter framerate: float; default value: 60.0

Frame rate

Parameter res_w: int; default value: 320

Resolution (width)

Parameter res_h: int; default value: 200

Resolution (height)

✎

No subscriptions defined.

✎

Publisher image_compressed: topic ~image/compressed" (CompressedImage)

Missing description for entry “image_compressed”.

Node decoder_node

✎🔗

(Generated from configuration decoder_node.easy_node.yaml.)

A node that decodes a compressed image into a regular image. This is useful so that multiple nodes that need to use the image do not do redundant computaon.

✎

Parameter publish_freq: float; default value: 1.0

Frequency at which to publish (Hz).

✎

Subscription compressed_image: topic ~compressed_image (CompressedImage)

The image to decode.

Subscription switch: topic ~switch (BoolStamped)

Switch to turn on or off. The node starts as active.

✎

Publisher raw: topic ~image/raw (Image)

The decoded image.

Node img_process_node

✎🔗

(Generated from configuration img_process_node.easy_node.yaml.)

Apparently, a template, or a node never finished.

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Node cam_info_reader_node

✎🔗

(Generated from configuration cam_info_reader_node.easy_node.yaml.)

Publishes a CameraInfo message every time it receives an image.

✎

Parameter config: str; default value: 'baseline'

Missing description for entry “config”.

Parameter cali_file_name: str; default value: 'default'

Missing description for entry “cali_file_name”.

Parameter image_type: str; default value: 'compressed'

Missing description for entry “image_type”.

✎

Subscription compressed_image: topic ~compressed_image (CompressedImage)

If image_type is “compressed” then it’s CompressedImage, otherwise Image.

✎

Publisher camera_info: topic ~camera_info (CameraInfo)

Missing description for entry “camera_info”.

Because of mathjax bug

Packages - Lane control

✎🔗

to write

Because of mathjax bug
Because of mathjax bug

Package anti_instagram

✎🔗

✎

Link to package on Github

✎

mfe

Mack

✎

Add a description of package anti_instagram in package.xml.

Node anti_instagram_node

✎🔗

(Generated from configuration anti_instagram_node.easy_node.yaml.)

Missing node description in anti_instagram_node.easy_node.yaml.

✎

Parameter publish_corrected_image: bool; default value: False

Whether to compute and publish the corrected image.

✎

Subscription image: topic ~uncorrected_image (CompressedImage)

This is the compressed image to read.

Subscription click: topic ~click (BoolStamped)

Activate the calibration phase with this switch.

✎

Publisher image: topic ~corrected_image (Image)

The corrected image.

Publisher health: topic ~colorSegment (AntiInstagramHealth)

The health of the process.

Publisher transform: topic ~transform (AntiInstagramTransform)

The computed transform.

Because of mathjax bug
Because of mathjax bug

Package complete_image_pipeline

✎🔗

✎

Link to package on Github

✎

Mack

✎

This package contains functions that allow to call the entire message pipeline programmatically, as a Python function, without having to instantiate any ROS node.

This is useful for regression tests, and quick tests with new logs.

Basic Markduck guide

✎🔗

The Duckiebook is written in Markduck, a Markdown dialect.

It supports many features that make it possible to create publication-worthy materials.

Figures

✎🔗

For any element, adding an attribute called figure-id with value fig:figure ID or tab:table ID will create a figure that wraps the element.

For example:

<div figure-id="fig:figure ID">
    figure content
</div>

It will create HMTL of the form:

<div id='fig:code-wrap' class='generated-figure-wrap'>
    <figure id='fig:figure ID' class='generated-figure'>
        <div>
            figure content
        </div>
    </figure>
</div>

To add a caption, add an attribute figure-caption:

<div figure-id="fig:figure ID" figure-caption="This is my caption">
    figure content
</div>

Alternatively, you can put anywhere an element figcaption with ID figure id:caption:

<element figure-id="fig:figure ID">
    figure content
</element>

<figcaption id='fig:figure ID:caption'>
    This the caption figure.
</figcaption>

To refer to the figure, use an empty link:

Please see [](#fig:figure ID).

The code will put a reference to “Figure XX”.

Linking to documentation

✎🔗

Establishing names of headers

✎🔗

You give IDs to headers using the format:

### header title {#topic ID}

For example, for this subsection, we have used:

### Establishing names of headers {#establishing}

With this, we have given this header the ID “establishing”.

Basic Markduck guide

✎🔗

The Duckiebook is written in Markduck, a Markdown dialect.

It supports many features that make it possible to create publication-worthy materials.

Figures

✎🔗

For any element, adding an attribute called figure-id with value fig:figure ID or tab:table ID will create a figure that wraps the element.

For example:

<div figure-id="fig:figure ID">
    figure content
</div>

It will create HMTL of the form:

<div id='fig:code-wrap' class='generated-figure-wrap'>
    <figure id='fig:figure ID' class='generated-figure'>
        <div>
            figure content
        </div>
    </figure>
</div>

To add a caption, add an attribute figure-caption:

<div figure-id="fig:figure ID" figure-caption="This is my caption">
    figure content
</div>

Alternatively, you can put anywhere an element figcaption with ID figure id:caption:

<element figure-id="fig:figure ID">
    figure content
</element>

<figcaption id='fig:figure ID:caption'>
    This the caption figure.
</figcaption>

To refer to the figure, use an empty link:

Please see [](#fig:figure ID).

The code will put a reference to “Figure XX”.

Linking to documentation

✎🔗

Establishing names of headers

✎🔗

You give IDs to headers using the format:

### header title {#topic ID}

For example, for this subsection, we have used:

### Establishing names of headers {#establishing}

With this, we have given this header the ID “establishing”.

Package ground_projection

✎🔗

✎

Link to package on Github

✎

Changhyun Choi

Mack

✎

Add a description of package ground_projection in package.xml.

Node ground_projection_node

✎🔗

(Generated from configuration ground_projection.easy_node.yaml.)

node description for ground_projection

✎

No parameters defined.

✎

No subscriptions defined.

✎

No publishers defined.

Because of mathjax bug
Because of mathjax bug

Package lane_control

✎🔗

✎

Link to package on Github

✎

steven

Shih-Yuan Liu

Liam Paull

✎

Add a description of package lane_control in package.xml.

lane_controller_node

✎🔗

(Generated from configuration lane_controller_node.easy_node.yaml.)

there is some very funny business inside. It appears that k_d and k_theta are switched around.

✎

Parameter v_bar: float

Nominal linear velocity (m/s).

Parameter k_theta: float

Proportional gain for $\theta$.

Parameter k_d: float

Propertional gain for $d$.

Parameter d_thres: float

Cap for error in $d$.

Parameter theta_thres: float

Maximum desired $\theta$.

Parameter d_offset: float

A configurable offset from the lane position.

✎

Subscription lane_reading: topic ~lane_pose (LanePose)

Missing description for entry “lane_reading”.

✎

Publisher car_cmd: topic ~car_cmd (Twist2DStamped)

Missing description for entry “car_cmd”.

Because of mathjax bug
Because of mathjax bug

Package lane_filter

✎🔗

Liam

✎

Link to package on Github

✎

Liam Paull

✎

Add a description of package lane_filter in package.xml.

lane_filter_node

✎🔗

(Generated from configuration lane_filter_node.easy_node.yaml.)

Missing node description in lane_filter_node.easy_node.yaml.

✎

Parameter mean_d_0: float; default value: 0.0

Missing description for entry “mean_d_0”.

Parameter mean_phi_0: float; default value: 0.0

Missing description for entry “mean_phi_0”.

Parameter sigma_d_0: float; default value: 0.0

Missing description for entry “sigma_d_0”.

Parameter sigma_phi_0: float; default value: 0.0

Missing description for entry “sigma_phi_0”.

Parameter delta_d: float; default value: 0.02

(meters)

Parameter delta_phi: float; default value: 0.0

(radians)

Parameter d_max: float; default value: 0.5

Missing description for entry “d_max”.

Parameter d_min: float; default value: -0.7

Missing description for entry “d_min”.

Parameter phi_min: float; default value: -1.5707

Missing description for entry “phi_min”.

Parameter phi_max: float; default value: 1.5707

Missing description for entry “phi_max”.

Parameter cov_v: float; default value: 0.5

Linear velocity “input”.

XXX which units?

Parameter cov_omega: float; default value: 0.01

Angular velocity “input”.

XXX which units?

Parameter linewidth_white: float; default value: 0.04

Missing description for entry “linewidth_white”.

Parameter linewidth_yellow: float; default value: 0.02

Missing description for entry “linewidth_yellow”.

Parameter lanewidth: float; default value: 0.4

Missing description for entry “lanewidth”.

Parameter min_max: float; default value: 0.3

Expressed in nats.

Parameter use_distance_weighting: bool; default value: False

For use of distance weighting (dw) function.

Parameter zero_val: float; default value: 1.0

Missing description for entry “zero_val”.

Parameter l_peak: float; default value: 1.0

Missing description for entry “l_peak”.

Parameter peak_val: float; default value: 10.0

Missing description for entry “peak_val”.

Parameter l_max: float; default value: 2.0

Missing description for entry “l_max”.

Parameter use_max_segment_dist: bool; default value: False

For use of maximum segment distance.

Parameter max_segment_dist: float; default value: 1.0

For use of maximum segment distance.

Parameter use_min_segs: bool; default value: False

For use of minimum segment count.

Parameter min_segs: int; default value: 10

For use of minimum segment count.

Parameter use_propagation: bool; default value: False

For propagation.

Parameter sigma_d_mask: float; default value: 0.05

Missing description for entry “sigma_d_mask”.

Parameter sigma_phi_mask: float; default value: 0.05

Missing description for entry “sigma_phi_mask”.

✎

Subscription velocity: topic ~velocity (Twist2DStamped)

Missing description for entry “velocity”.

Subscription segment_list: topic ~segment_list (SegmentList)

Missing description for entry “segment_list”.

✎

Publisher lane_pose: topic ~lane_pose (LanePose)

Missing description for entry “lane_pose”.

Publisher belief_img: topic ~belief_img (Image)

Missing description for entry “belief_img”.

Publisher entropy: topic ~entropy (Float32)

Missing description for entry “entropy”.

Publisher in_lane: topic ~in_lane (BoolStamped)

Missing description for entry “in_lane”.

Publisher switch: topic ~switch (BoolStamped)

Missing description for entry “switch”.

Because of mathjax bug
Because of mathjax bug

Package line_detector2

✎🔗

✎

Link to package on Github

✎

Andrea Censi (maintainer)

✎

A re-implementation of the line detector node.

This is a re-implementation of the package line_detector using the new facilities provided by easy_node.

line_detector_node2

✎🔗

(Generated from configuration line_detector_node2.easy_node.yaml.)

This is a rewriting of line_detector_node using the EasyNode framework.

✎

Parameter verbose: bool; default value: True

Whether the node is verbose or not. If set to True, the node will write timing statistics to the log.

Parameter img_size: not known

Missing description for entry “img_size”.

Parameter top_cutoff: int

This parameter decides how much of the image we should cut off. This is a performance improvement.

Parameter line_detector: str

This is the instance of line_detector to use.

✎

Subscription image: topic ~image (CompressedImage)

This is the compressed image to read. Note that it takes a long time to simply decode the image JPG.

The data is processed asynchronously in a different thread.

Subscription transform: topic ~transform (AntiInstagramTransform)

The anti-instagram transform to apply. See Unit Q-1 - Package anti_instagram.

Subscription switch: topic ~switch (BoolStamped)

This is a switch that allows to control the activity of this node. If the message is true, the node becomes active. If false, it switches off. The node starts as active.

✎

Publisher edge: topic ~edge (Image)

Missing description for entry “edge”.

Publisher color_segment: topic ~colorSegment (Image)

Missing description for entry “color_segment”.

Publisher segment_list: topic ~segment_list (SegmentList)

Missing description for entry “segment_list”.

Publisher image_with_lines: topic ~image_with_lines (Image)

Missing description for entry “image_with_lines”.

Because of mathjax bug
Because of mathjax bug

Package line_detector

✎🔗

✎

Link to package on Github

✎

Hang Zhao

Mack

✎

Add a description of package line_detector in package.xml.

This package is being replaced by the cleaned-up version, line_detector2. Do not write documentation here.

However, at the moment, all the launch files still call this one.

Because of mathjax bug

Packages - Indefinite navigation

✎🔗

to write

Because of mathjax bug
Because of mathjax bug

AprilTags library

✎🔗

✎

Link to package on Github

✎

Michael Kaess

Hordur Johannson

Mitchell Wills

✎

A catkin version of the C++ apriltags library

Detect April tags (2D bar codes) in images; reports unique ID of each detection, and optionally its position and orientation relative to a calibrated camera.

See examples/apriltags_demo.cpp for a simple example that detects April tags (see tags/pdf/tag36h11.pdf) in laptop or webcam images and marks any tags in the live image.

Ubuntu dependencies: sudo apt-get install subversion cmake libopencv-dev libeigen3-dev libv4l-dev

Mac dependencies: sudo port install pkgconfig opencv eigen3

Uses the pods build system in connection with cmake, see: http://sourceforge.net/p/pods/

Michael Kaess October 2012

AprilTags were developed by Professor Edwin Olson of the University of Michigan. His Java implementation is available on this web site: http://april.eecs.umich.edu.

Olson’s Java code was ported to C++ and integrated into the Tekkotsu framework by Jeffrey Boyland and David Touretzky.

See this Tekkotsu wiki article for additional links and references: http://wiki.tekkotsu.org/index.php/AprilTags

This C++ code was further modified by Michael Kaess (kaess@mit.edu) and Hordur Johannson (hordurj@mit.edu) and the code has been released under the LGPL 2.1 license.

  • converted to standalone library
  • added stable homography recovery using OpenCV
  • robust tag code table that does not require a terminating 0 (omission results in false positives by illegal codes being accepted)
  • changed example tags to agree with Ed Olson’s Java version and added all his other tag families
  • added principal point as parameter as in original code - essential for homography
  • added some debugging code (visualization using OpenCV to show intermediate detection steps)
  • added fast approximation of arctan2 from Ed’s original Java code
  • using interpolation instead of homography in Quad: requires less homography computations and provides a small improvement in correct detections

todo: - significant speedup could be achieved by performing image operations using OpenCV (Gaussian filter, but also operations in TagDetector.cc) - replacing arctan2 by precomputed lookup table - converting matrix operations to Eigen (mostly for simplifying code, maybe some speedup)

Because of mathjax bug
Because of mathjax bug

Package apriltags_ros

✎🔗

AprilTags for ROS.

Build Status

Resource error
I will not embed remote files, such as https://api.travis-ci.org/RIVeR-Lab/apriltags_ros.png:

✎

Link to package on Github

✎

Mitchell Wills

Mitchell Wills

✎

A package that provides a ROS wrapper for the apriltags C++ package

Because of mathjax bug
Because of mathjax bug

Package fsm

✎🔗

✎

Link to package on Github

✎

Michael Misha Novitzky

Mack

✎

The finite state machine coordinates the modes of the car. The fsm package consists of two nodes, namely fsm_node and logic_gate_node.

Node fsm_node

✎🔗

(Generated from configuration fsm_node.easy_node.yaml.)

fsm_node

This node handles the state transitions based on the defined state transition events. Below is a summary of the basic functionality of the fsm_node.

  • Each state is a mode that the Duckiebot can be in
  • Each state has corresponding state transitions triggered by events
  • Each event is triggered by a certain value of a certain topic message
  • In each state, certain nodes are active
  • Each node affected by the state machine can be switched active/inactive by a ~/switch topic

The current state is published to the fsm_node/mode topic. For each state, there is a list of nodes which should be active, which are switched by means of node_name/switch topics.

The FSM node publishes on many topics according to the configuration:

for node_name, topic_name in nodes.items():
    self.pub_dict[node_name] = rospy.Publisher(topic_name, BoolStamped, ...)

where nodes.items() is a list of all nodes affected by the FSM, and the topic_name is node_name/switch. The relevant nodes then subscribe to ~/switch, and toggle their behaviour based on the value of the switch. Nodes can also subscribe to the fsm_node/mode topic if they need to change their behaviour based on the state. An example of how a node named ExampleNode can handle this is shown below:

class ExampleNode(object):
    def \__init\__(self):
    ...
    self.sub_switch = rospy.Subscriber("~switch",BoolStamped, self.cbSwitch, queue_size=1)
    self.sub_fsm_mode = rospy.Subscriber("fsm_node/mode",FSMState, self.cbMode, queue_size=1)
    self.active = True
    self.mode = None

    def cbSwitch(self,switch_msg):
        self.active = switch_msg.data # True or False

    def cbMode(self,switch_msg):
        self.mode = switch_msg.state # String of current FSM state

    def someOtherFunc(self, msg):
        if not self.active:
            return
        # else normal functionality
        ...
        if self.mode == "LANE_FOLLOWING":
            ...
        if self.mode == "INTERSECTION_CONTROL":
            ...
✎

Parameter states: dict; default value: ``

States are the modes that the system can be in. Each state has corresponding events (which trigger transitions to specific states), as well as a list of active nodes in the current state.

Parameter nodes: dict; default value: ``

These are the nodes which are affected by the FSM, and also define the ~/switch topics to switch them between active and inactive.

Parameter global_transitions: dict; default value: ``

These are the state transition events (and corresponding topic) that can be triggered from all states.

Parameter initial_state: str; default value: 'LANE_FOLLOWING'

This is the initial state that the FSM will be in upon launch of the node.

Parameter events: dict; default value: ``

These are the events and the corresponding topics (and message values) which trigger them, which allow for transitions between states.

✎

No subscriptions defined.

✎

Publisher mode: topic ~mode (FSMState)

This topic gives the current state of the FSM, and can have values from a set of strings indicating the possible state names.

FSM diagram

Node logic_gate_node

✎🔗

(Generated from configuration logic_gate_node.easy_node.yaml.)

logic_gate_node

This node handles AND and OR logic gates of events for state transitions. Below is a summary of the basic functionality of the logic_gate_node.

  • Logic AND and OR gates can be defined
  • For each gate, the input events (and their corresponding topics) are defined
  • The logic_gate_node subscribes to all of these input event topics
  • When an input topic is published, the logic_gate_node checks whether the AND or OR gate is satisfied
  • If the gate is satisfied, the node publishes True on the ~/gate_name topic, else it publishes False

The logic gate node publishes on many topics according to the configuration:

for gate_name, gate_dict in self.gates_dict.items():
    output_topic_name = gate_dict["output_topic"]
    self.pub_dict[gate_name] = rospy.Publisher(output_topic_name, BoolStamped, queue_size=1)

where gate_dict.items() is a dictionary of all gates, and output_topic_name is ~/gate_name. The fsm_node then subscribes to logic_gate_node/*, where each gate_name corresponds to a state transition event.

✎

Parameter events: dict; default value: ``

These are all the events and corresponding topics (and trigger values) which are inputs to a logic gate event.

Parameter gates: dict; default value: ``

These are the logic gate events. Each gate has a gate_type (AND or OR), input events, and an output topic.

✎

No subscriptions defined.

✎

No publishers defined.

Because of mathjax bug
Because of mathjax bug

Package indefinite_navigation

✎🔗

✎

Link to package on Github

✎

Liam Paull

✎

Add a description of package indefinite_navigation in package.xml.

Because of mathjax bug
Because of mathjax bug

Package intersection_control

✎🔗

✎

Link to package on Github

✎

Michael Misha Novitzky

Mack

✎

The intersection_control package implements a dead reckoning controller until we do something smarter.

Because of mathjax bug
Because of mathjax bug

Package stop_line_filter

✎🔗

✎

Link to package on Github

✎

Liam Paull

✎

Add a description of package stop_line_filter in package.xml.

Because of mathjax bug

Packages - Localization and planning

✎🔗

to write

Because of mathjax bug
Because of mathjax bug

Package duckietown_description

✎🔗

✎

Link to package on Github

✎

Teddy Ort

Mack

✎

Add a description of package duckietown_description in package.xml.

Because of mathjax bug
Because of mathjax bug

Package localization

✎🔗

✎

Link to package on Github

✎

teddy

Mack

✎

Add a description of package localization in package.xml.

Because of mathjax bug

Packages - Coordination

✎🔗

to write

Because of mathjax bug
Because of mathjax bug

Package led_detection

✎🔗

✎

Link to package on Github

✎

Andrea Censi

✎

Add a description of package led_detection in package.xml.

add authors

Package led_emitter

✎🔗

✎

Link to package on Github

✎

Andrea Censi

✎

description for led_emitter package.

Package led_interpreter

✎🔗

✎

Link to package on Github

✎

qlai

Mack

✎

Add a description of package led_interpreter in package.xml.

Because of mathjax bug
Because of mathjax bug

Package led_joy_mapper

✎🔗

✎

Link to package on Github

✎

lapentab

Mack

✎

Add a description of package led_joy_mapper in package.xml.

Because of mathjax bug
Because of mathjax bug

Package rgb_led

✎🔗

✎

Link to package on Github

✎

Dmitry Yershov

Mack

✎

Add a description of package rgb_led in package.xml.

Package traffic_light

✎🔗

✎

Link to package on Github

✎

Mack

✎

Add a description of package traffic_light in package.xml.

Because of mathjax bug

Packages - Additional functionality

✎🔗

to write

Because of mathjax bug
Because of mathjax bug

Package mdoap

✎🔗

✎

Link to package on Github

✎

Catherine Liu

Sam

Ari

Mack

✎

The mdoap package which handles object recognition and uses ground projection to estimate distances of objects for duckie safety. Also contains controllers for avoiding said obstacles

Because of mathjax bug
Because of mathjax bug

Package parallel_autonomy

✎🔗

✎

Link to package on Github

✎

Liam Paull

✎

description for the parallel autonomy package

Because of mathjax bug
Because of mathjax bug

Package vehicle_detection

✎🔗

✎

Link to package on Github

✎

Andrea Tacchetti

Mack

✎

Add a description of package vehicle_detection in package.xml.

Because of mathjax bug

Packages - Templates

✎🔗

These are templates.

Because of mathjax bug
Because of mathjax bug

Package pkg_name

✎🔗

✎

Link to package on Github

✎

Shih-Yuan Liu

Andrea Censi

✎

The package pkg_name is a template for ROS packages.

For the tutorial, see Unit K-6 - Minimal ROS node - pkg_name.

Package rostest_example

✎🔗

✎

Link to package on Github

✎

Teddy Ort

Mack

✎

Add a description of package rostest_example in package.xml.

Because of mathjax bug

Packages - Convenience

✎🔗

These packages are convenience packages that group together launch files and tests.

Because of mathjax bug
Because of mathjax bug

Package duckie_rr_bridge

✎🔗

✎

Link to package on Github

✎

greg

Mack

✎

The duckie_rr_bridge package. Creates a Robot Raconteur Service to drive the Duckiebot.

Because of mathjax bug
Because of mathjax bug

Package duckiebot_visualizer

✎🔗

✎

Link to package on Github

✎

Shih-Yuan Liu

Mack

✎

Add a description of package duckiebot_visualizer in package.xml.

Node duckiebot_visualizer

✎🔗

(Generated from configuration duckiebot_visualizer.easy_node.yaml.)

Missing node description in duckiebot_visualizer.easy_node.yaml.

✎

Parameter veh_name: str; default value: 'megaman'

Missing description for entry “veh_name”.

✎

Subscription seg_list: topic ~segment_list (SegmentList)

Missing description for entry “seg_list”.

✎

Publisher pub_seg_list: topic ~segment_list_markers (MarkerArray)

Missing description for entry “pub_seg_list”.

Because of mathjax bug
Because of mathjax bug

Package duckietown_demos

✎🔗

✎

Link to package on Github

✎

Liam Paull

✎

description of duckietown_demos

Because of mathjax bug
Because of mathjax bug

Package duckietown_unit_test

✎🔗

✎

Link to package on Github

✎

Mack

✎

The duckietown_unit_test meta package contains all the unit test launch files for Duckietown.

Because of mathjax bug
Because of mathjax bug

Package veh_coordinator

✎🔗

✎

Link to package on Github

✎

Michal Cap

Mack

✎

The simple vehicle coordination package

I think is used to fake the vehicle coordination for the FSM.

Because of mathjax bug

Packages - Deep Learning

✎🔗

to write

Because of mathjax bug
Jeffrey Dean and Sanjay Ghemawat. Mapreduce: a flexible data processing tool. Commun. ACM, 53(1):72–77, 2010.    DOI  http  Gregory Dudek and Michael Jenkin. Computational principles of mobile robotics. Cambridge university press, 2010.    Rached Dhaouadi and A. Abu Hatab. Dynamic modeling of differential-drive mobile robots using lagrange and newton-euler methodologies: A unified framework. Advances in Robotics & Automation, 2(2):1–7, 2013.   

Last modified